Sep 10 12:28:18.502373 ip-10-0-140-169 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 12:28:18.502384 ip-10-0-140-169 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 12:28:18.502391 ip-10-0-140-169 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 12:28:18.502533 ip-10-0-140-169 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 12:28:28.665031 ip-10-0-140-169 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 12:28:28.665047 ip-10-0-140-169 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot da43d8c014ff47f59086ec08cb7181f4 --
Sep 10 12:30:24.230764 ip-10-0-140-169 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 12:30:24.645833 ip-10-0-140-169 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:30:24.645833 ip-10-0-140-169 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 12:30:24.645833 ip-10-0-140-169 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:30:24.645833 ip-10-0-140-169 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 12:30:24.645833 ip-10-0-140-169 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:30:24.648575 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.648491    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 12:30:24.655231 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655213    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.655231 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655227    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.655231 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655231    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.655231 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655234    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.655231 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655237    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655240    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655243    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655246    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655249    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655253    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655255    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655260    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655263    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655267    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655270    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655273    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655277    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655286    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655289    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655292    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655295    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655298    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655300    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.655408 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655303    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655306    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655308    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655311    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655314    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655316    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655319    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655322    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655325    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655328    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655330    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655332    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655335    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655338    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655340    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655344    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655347    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655349    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655352    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655356    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.655934 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655360    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655363    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655366    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655369    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655372    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655374    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655377    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655379    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655382    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655385    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655387    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655390    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655392    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655395    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655397    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655400    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655402    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655405    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655407    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655410    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.656417 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655414    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655417    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655419    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655422    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655424    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655427    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655429    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655433    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655436    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655438    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655441    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655443    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655446    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655448    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655451    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655455    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655458    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655460    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655463    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655465    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.656925 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655468    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655470    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655485    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655489    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655491    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655494    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655496    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655499    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655501    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655504    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655507    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655510    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655512    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655515    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655518    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.655521    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656226    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656233    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656236    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656239    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.657439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656242    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656245    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656248    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656250    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656253    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656256    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656258    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656261    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656268    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656271    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656274    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656277    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656279    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656282    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656285    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656287    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656290    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656292    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656295    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656298    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.657939 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656300    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656303    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656305    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656308    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656310    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656312    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656315    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656318    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656320    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656325    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656334    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656338    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656341    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656344    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656346    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656349    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656352    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656354    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656358    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656360    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.658489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656363    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656367    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656370    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656373    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656375    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656378    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656380    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656383    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656385    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656388    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656391    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656393    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656396    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656398    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656401    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656404    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656406    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656409    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656411    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656414    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.658985 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656416    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656419    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656422    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656426    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656429    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656431    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656435    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656440    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656445    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656448    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656450    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656453    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656458    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656461    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656464    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656467    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656469    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656492    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656495    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.659466 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656508    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656511    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656513    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656516    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656519    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656521    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656524    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656527    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656529    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656532    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656534    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656537    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656539    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656542    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656544    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656547    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656900    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656904    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656907    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.659965 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656910    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656913    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656915    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656918    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656921    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656924    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656926    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656932    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656936    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656940    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656942    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656945    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656948    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656951    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656953    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656956    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656959    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656962    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656964    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.660436 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656966    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656969    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656972    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656975    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656977    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656980    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656982    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656985    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656987    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656990    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656993    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656996    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.656998    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657001    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657004    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657007    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657010    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657012    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657015    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657018    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.660909 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657021    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657023    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657027    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657029    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657032    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657035    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657037    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657039    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657042    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657044    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657047    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657050    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657052    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657055    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657058    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657060    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657064    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657066    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657069    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657071    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.661400 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657074    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657076    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657079    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657082    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657084    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657087    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657090    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657093    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657095    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657098    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657100    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657103    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657106    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657109    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657111    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657114    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657117    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657121    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657125    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657129    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.661891 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657131    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657134    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657137    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657140    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657142    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657145    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657147    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657150    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657153    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657156    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657158    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657161    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657164    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657166    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657169    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657172    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657176    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657257    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657304    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657311    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657319    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 12:30:24.662379 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657323    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657326    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657330    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657334    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657338    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657341    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657344    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657349    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657352    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657355    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657358    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657361    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657364    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657367    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657371    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657374    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657376    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657379    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657383    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657386    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657389    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657392    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657396    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657399    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 12:30:24.662918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657401    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657404    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657407    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657410    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657417    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657419    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657422    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657425    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657428    2577 flags.go:64] FLAG: --enable-server="true"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657431    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657435    2577 flags.go:64] FLAG: --event-burst="100"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657438    2577 flags.go:64] FLAG: --event-qps="50"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657441    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657444    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657447    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657451    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657455    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657458    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657461    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657464    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657466    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657469    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657483    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657486    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657489    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 12:30:24.663555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657492    2577 flags.go:64] FLAG: --feature-gates=""
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657495    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657499    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657502    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657506    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657509    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657511    2577 flags.go:64] FLAG: --help="false"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657514    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-140-169.ec2.internal"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657517    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657520    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657523    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657526    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657529    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657532    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657535    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657538    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657540    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657545    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657548    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657550    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657553    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657556    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657559    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657562    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 12:30:24.664215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657565    2577 flags.go:64] FLAG: --lock-file=""
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657568    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657571    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657574    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657579    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657582    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657585    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657587    2577 flags.go:64] FLAG: --logging-format="text"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657590    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657594    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657596    2577 flags.go:64] FLAG: --manifest-url=""
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657599    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657604    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657607    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657612    2577 flags.go:64] FLAG: --max-pods="110"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657615    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657618    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657621    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657624    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657627    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657630    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657633    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657640    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657643    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657645    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 12:30:24.664813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657650    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657653    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657658    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657661    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657664    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657667    2577 flags.go:64] FLAG: --port="10250"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657670    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657673    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b4cf41c8a630e7ea"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657677    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657680    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657683    2577 flags.go:64] FLAG: --register-node="true"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657685    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657689    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657692    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657694    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657698    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657701    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657705    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657707    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657711    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657713    2577 flags.go:64] FLAG: --runonce="false"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657716    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657719    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657722    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657725    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 12:30:24.665686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657728    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657731    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657734    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657737    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657740    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657742    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657745    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657749    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657752    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657755    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657758    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657763    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657766    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657769    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657773    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657776    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657779    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657782    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657784    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657787    2577 flags.go:64] FLAG: --v="2"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657791    2577 flags.go:64] FLAG: --version="false"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657795    2577 flags.go:64] FLAG: --vmodule=""
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657798    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.657801    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 12:30:24.666428 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657917    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657921    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657924    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657928    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657930    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657933    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657936    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657938    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657941    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657944    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657946    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657949    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657952    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657954    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657957    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657961    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657964    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657966    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657969    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657971    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.667045 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657974    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657977    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657979    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657982    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657984    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657989    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657992    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657995    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.657998    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658001    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658004    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658006    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658009    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658011    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658016    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658018    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658021    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658024    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658026    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.667585 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658029    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658032    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658034    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658038    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658042    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658044    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658047    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658051    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658056    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658059    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658062    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658065    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658068    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658071    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658074    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658076    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658079    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658082    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658084    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.668084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658087    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658090    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658093    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658095    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658098    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658100    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658103    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658105    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658109    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658112    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658114    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658117    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658119    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658122    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658125    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658128    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658130    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658133    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658135    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658138    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.668570 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658141    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658144    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658147    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658149    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658153    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658156    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658159    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658161    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658164    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658166    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658169    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658172    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658174    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658177    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658188    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658191    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658193    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658196    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658198    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658201    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.669052 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.658205    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.659576    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.667788    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.667804    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.667823    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.667829    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667884    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667888    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667892    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667895    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667898    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667900    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667904    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667906    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667909    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.669582 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667912    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667914    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667917    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667920    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667922    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667925    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667928    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667930    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667933    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667935    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667938    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667941    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667944    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667946    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667949    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667951    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667955    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667957    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667959    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667962    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.669957 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667965    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667968    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667971    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667973    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667976    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667978    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667981    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667984    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667986    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667989    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667991    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667994    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667996    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.667998    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668001    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668004    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668006    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668009    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668011    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668014    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.670450 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668016    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668019    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668022    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668024    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668026    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668029    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668031    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668034    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668037    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668041    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668045    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668048    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668051    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668054    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668057    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668060    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668062    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668065    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668067    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.671043 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668070    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668072    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668075    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668078    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668080    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668083    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668085    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668088    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668091    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668094    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668096    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668101    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668104    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668107    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668110    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668113    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668116    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668118    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668121    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.671698 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668124    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668126    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668129    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668132    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668134    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668137    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668140    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668143    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668145    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668149    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668153    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668156    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.668161    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668285    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668290    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.672167 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668293    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668296    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668299    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668301    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668304    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668307    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668309    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668312    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668314    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668317    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668324    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668327    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668329    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668333    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668337    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668340    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668343    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668346    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668349    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.672556 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668352    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668355    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668358    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668361    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668363    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668366    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668369    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668371    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668374    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668377    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668380    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668382    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668385    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668387    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668390    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668392    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668395    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668398    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668400    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.673016 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668403    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668405    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668408    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668411    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668413    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668416    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668419    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668421    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668424    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668426    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668429    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668433    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668436    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668439    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668442    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668445    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668447    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668450    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668453    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.673467 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668455    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668458    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668461    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668463    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668466    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668469    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668486    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668490    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668493    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668495    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668498    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668501    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668503    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668506    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668509    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668511    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668514    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668516    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668518    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668523    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.673931 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668527    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668530    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668533    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668536    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668539    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668542    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668545    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668547    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668550    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668552    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668555    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668557    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668560    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668563    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668566    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668568    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668571    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668574    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668576    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.674407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:24.668579    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.674886 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.668584    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:30:24.674886 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.668688    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 12:30:24.674886 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.671372    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 12:30:24.674886 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.672343    2577 server.go:1019] "Starting client certificate rotation"
Sep 10 12:30:24.674886 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.672442    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 12:30:24.674886 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.673268    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 12:30:24.674886 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.674694    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 12:30:24.675889 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.675872    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 12:30:24.686217 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.686199    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 10 12:30:24.689099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.689082    2577 log.go:25] "Validated CRI v1 image API"
Sep 10 12:30:24.689970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.689954    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 12:30:24.692832 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.692813    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bc4ed3b4-690e-4286-ae12-544561c036e8:/dev/nvme0n1p4 cefcca60-ad20-415a-bef0-ae1e8da563d5:/dev/nvme0n1p3]
Sep 10 12:30:24.692885 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.692832    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 12:30:24.699487 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.699366    2577 manager.go:216] Machine: {Timestamp:2026-09-10 12:30:24.698056133 +0000 UTC m=+0.344345431 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100297 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c334fa1e33e3f73367f0fe6156783 SystemUUID:ec2c334f-a1e3-3e3f-7336-7f0fe6156783 BootID:da43d8c0-14ff-47f5-9086-ec08cb7181f4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f5:59:e2:cb:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f5:59:e2:cb:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:fb:b6:5a:bc:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 12:30:24.699487 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.699466    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 12:30:24.699636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.699564    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 12:30:24.700511 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.700468    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 12:30:24.700636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.700513    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-169.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 12:30:24.701267 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.701257    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 12:30:24.701303 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.701268    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 12:30:24.701303 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.701276    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 12:30:24.701946 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.701935    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 12:30:24.703249 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.703235    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 12:30:24.706977 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.706964    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 12:30:24.707095 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707085    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 12:30:24.707136 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707115    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 12:30:24.707136 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707125    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 12:30:24.707136 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707142    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 12:30:24.707284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707152    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 10 12:30:24.707284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707160    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 12:30:24.707430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707408    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 12:30:24.707574 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707561    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 12:30:24.707629 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.707585    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 12:30:24.710126 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.710098    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 12:30:24.712312 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.712297    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 12:30:24.712369 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.712328    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 12:30:24.713234 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713213    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 12:30:24.713279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713241    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 12:30:24.713279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713247    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 12:30:24.713279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713252    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 12:30:24.713279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713262    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 12:30:24.713279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713267    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 12:30:24.713414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713282    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 12:30:24.713414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713288    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 12:30:24.713414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713295    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 12:30:24.713414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713301    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 12:30:24.713414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713309    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 12:30:24.713414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713315    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 12:30:24.713414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713341    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 12:30:24.713414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.713350    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 12:30:24.717308 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.717292    2577 server.go:1268] "Started kubelet"
Sep 10 12:30:24.717396 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.717380    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-169.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 12:30:24.717454 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.717375    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 12:30:24.717526 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.717421    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 12:30:24.717526 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.717497    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 12:30:24.717526 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.717511    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-169.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 12:30:24.717657 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.717544    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 12:30:24.718460 ip-10-0-140-169 systemd[1]: Started Kubernetes Kubelet.
Sep 10 12:30:24.719708 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.719692    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 12:30:24.723060 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.723040    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 12:30:24.726081 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.725039    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 12:30:24.727882 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.727864    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 12:30:24.728466 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.728451    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 12:30:24.728651 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.728597    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:24.729038 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.728061    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-169.ec2.internal.18d3f5841a0570d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-169.ec2.internal,UID:ip-10-0-140-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-169.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.717271252 +0000 UTC m=+0.363560549,LastTimestamp:2026-09-10 12:30:24.717271252 +0000 UTC m=+0.363560549,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-169.ec2.internal,}"
Sep 10 12:30:24.729177 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729120    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 12:30:24.729177 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729124    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 12:30:24.729177 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729152    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 12:30:24.729284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729235    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 12:30:24.729284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729245    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 12:30:24.729284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729259    2577 factory.go:55] Registering systemd factory
Sep 10 12:30:24.729284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729276    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 10 12:30:24.729507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729496    2577 factory.go:153] Registering CRI-O factory
Sep 10 12:30:24.729507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729509    2577 factory.go:223] Registration of the crio container factory successfully
Sep 10 12:30:24.729574 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729546    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 12:30:24.729574 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729562    2577 factory.go:103] Registering Raw factory
Sep 10 12:30:24.729574 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729571    2577 manager.go:1196] Started watching for new ooms in manager
Sep 10 12:30:24.729933 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.729922    2577 manager.go:349] Starting recovery of all containers
Sep 10 12:30:24.735684 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.735529    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-169.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 12:30:24.735684 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.735556    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 12:30:24.739282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.739269    2577 manager.go:354] Recovery completed
Sep 10 12:30:24.744299 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.744283    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:24.747032 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.747016    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bnxh7"
Sep 10 12:30:24.747968 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.747942    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:24.748038 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.747985    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:24.748038 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.748001    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:24.750549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.749648    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 12:30:24.750549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.749665    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 12:30:24.750549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.749684    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 12:30:24.750549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.749756    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:24.751039 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.750873    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-169.ec2.internal.18d3f5841bd9d379  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-169.ec2.internal,UID:ip-10-0-140-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-169.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-169.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.747967353 +0000 UTC m=+0.394256653,LastTimestamp:2026-09-10 12:30:24.747967353 +0000 UTC m=+0.394256653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-169.ec2.internal,}"
Sep 10 12:30:24.751249 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.751224    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:24.751378 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.751364    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:24.751430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.751392    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:24.756053 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.756024    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bnxh7"
Sep 10 12:30:24.762654 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.762591    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-169.ec2.internal.18d3f5841bda32be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-169.ec2.internal,UID:ip-10-0-140-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-169.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-169.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.747991742 +0000 UTC m=+0.394281041,LastTimestamp:2026-09-10 12:30:24.747991742 +0000 UTC m=+0.394281041,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-169.ec2.internal,}"
Sep 10 12:30:24.770956 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.770898    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-169.ec2.internal.18d3f5841bda6f29  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-169.ec2.internal,UID:ip-10-0-140-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-169.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-169.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.748007209 +0000 UTC m=+0.394296507,LastTimestamp:2026-09-10 12:30:24.748007209 +0000 UTC m=+0.394296507,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-169.ec2.internal,}"
Sep 10 12:30:24.781300 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.781220    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-169.ec2.internal.18d3f5841bd9d379\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-169.ec2.internal.18d3f5841bd9d379  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-169.ec2.internal,UID:ip-10-0-140-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-169.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-169.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.747967353 +0000 UTC m=+0.394256653,LastTimestamp:2026-09-10 12:30:24.751330974 +0000 UTC m=+0.397620275,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-169.ec2.internal,}"
Sep 10 12:30:24.792408 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.792335    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-169.ec2.internal.18d3f5841bda32be\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-169.ec2.internal.18d3f5841bda32be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-169.ec2.internal,UID:ip-10-0-140-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-169.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-169.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.747991742 +0000 UTC m=+0.394281041,LastTimestamp:2026-09-10 12:30:24.751379542 +0000 UTC m=+0.397668850,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-169.ec2.internal,}"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.800501    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-169.ec2.internal.18d3f5841bda6f29\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-169.ec2.internal.18d3f5841bda6f29  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-169.ec2.internal,UID:ip-10-0-140-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-169.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-169.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.748007209 +0000 UTC m=+0.394296507,LastTimestamp:2026-09-10 12:30:24.75139964 +0000 UTC m=+0.397688944,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-169.ec2.internal,}"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.829709    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.868145    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.869342    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.869390    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.869416    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.869459    2577 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.879171    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.896351    2577 policy_none.go:49] "None policy: Start"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.896368    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 12:30:24.898355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.896380    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 12:30:24.899221 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.899209    2577 policy_none.go:47] "Start"
Sep 10 12:30:24.927998 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.927978    2577 manager.go:337] "Starting Device Plugin manager"
Sep 10 12:30:24.928090 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.928020    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 12:30:24.928090 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.928033    2577 server.go:87] "Starting device plugin registration server"
Sep 10 12:30:24.928364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.928347    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 12:30:24.928437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.928362    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 12:30:24.928520 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.928502    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 12:30:24.928606 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.928589    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 12:30:24.928660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.928610    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 12:30:24.929246 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.929228    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 12:30:24.929319 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.929275    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:24.943522 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.943503    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-169.ec2.internal\" not found" node="ip-10-0-140-169.ec2.internal"
Sep 10 12:30:24.969577 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.969548    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal"]
Sep 10 12:30:24.969661 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.969646    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:24.971005 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.970986    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:24.971090 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.971011    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:24.971090 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.971031    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:24.973288 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.973276    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:24.973408 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.973393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:24.973450 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.973422    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:24.974076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.974060    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:24.974076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.974067    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:24.974217 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.974089    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:24.974217 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.974091    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:24.974217 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.974100    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:24.974217 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.974105    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:24.976393 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.976379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:24.976455 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.976403    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:24.977158 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.977143    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:24.977238 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.977170    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:24.977238 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:24.977184    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:24.997962 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:24.997945    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-169.ec2.internal\" not found" node="ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.001997 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:25.001982    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-169.ec2.internal\" not found" node="ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.029233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.029212    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:25.029973 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.029953    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:25.030067 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.029984    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:25.030067 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.030001    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:25.030067 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.030028    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.030249 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.030233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f88650c6069898e4f5d2a7851171af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"f2f88650c6069898e4f5d2a7851171af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.030313 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.030255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f88650c6069898e4f5d2a7851171af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"f2f88650c6069898e4f5d2a7851171af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.030355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.030309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ad92d93c55b1bd4bbad87ef6544190d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-169.ec2.internal\" (UID: \"9ad92d93c55b1bd4bbad87ef6544190d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.038006 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.037992    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.038061 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:25.038010    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-169.ec2.internal\": node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:25.051571 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:25.051553    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:25.130407 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.130383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f88650c6069898e4f5d2a7851171af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"f2f88650c6069898e4f5d2a7851171af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.130521 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.130411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f88650c6069898e4f5d2a7851171af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"f2f88650c6069898e4f5d2a7851171af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.130521 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.130433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ad92d93c55b1bd4bbad87ef6544190d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-169.ec2.internal\" (UID: \"9ad92d93c55b1bd4bbad87ef6544190d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.130521 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.130504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ad92d93c55b1bd4bbad87ef6544190d-config\") pod \"kube-apiserver-proxy-ip-10-0-140-169.ec2.internal\" (UID: \"9ad92d93c55b1bd4bbad87ef6544190d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.130640 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.130515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f88650c6069898e4f5d2a7851171af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"f2f88650c6069898e4f5d2a7851171af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.130640 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:25.130507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f88650c6069898e4f5d2a7851171af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal\" (UID: \"f2f88650c6069898e4f5d2a7851171af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:25.152538 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:25.152498    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:26.377352 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.377332    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:26.424501 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.424473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:26.429826 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.429461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:26.477407 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.477383    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:26.577871 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.577813    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:26.678394 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.678364    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-169.ec2.internal\" not found"
Sep 10 12:30:26.686309 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.686289    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:26.753571 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.753536    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:26.767492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.767474    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 12:30:26.769615 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.769590    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"
Sep 10 12:30:26.780302 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.780283    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 12:30:26.796429 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.796405    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 12:30:26.796524 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.796508    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 12:30:26.831759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.831697    2577 apiserver.go:52] "Watching apiserver"
Sep 10 12:30:26.848358 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.848334    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 12:30:26.849343 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.849318    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-fwl5z","openshift-multus/multus-lfc2h","kube-system/global-pull-secret-syncer-n4575","kube-system/konnectivity-agent-6zsm8","kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal","openshift-image-registry/node-ca-rrqld","openshift-multus/network-metrics-daemon-dj27m","openshift-network-diagnostics/network-check-target-btxk6","openshift-network-operator/iptables-alerter-9pvpv","openshift-ovn-kubernetes/ovnkube-node-ftz99","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj","openshift-cluster-node-tuning-operator/tuned-h25qd","openshift-dns/node-resolver-59cz5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal"]
Sep 10 12:30:26.851914 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.851897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.852163 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.852116    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 12:30:26.853884 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.853869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.854010 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.853992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 12:30:26.854319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.854302    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 12:30:26.854367 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.854306    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 12:30:26.854486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.854473    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp9zq\""
Sep 10 12:30:26.854745 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.854732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 12:30:26.854798 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.854754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 12:30:26.855932 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.855913    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gsnkx\""
Sep 10 12:30:26.856007 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.855989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.856061 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.856041    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:26.856118 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856058    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 12:30:26.856746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.856802 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-system-cni-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.856802 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-cnibin\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.856802 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-cni-bin\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.856802 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-socket-dir-parent\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.856918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/646110b4-56d3-44b3-b75c-3fb2883b274b-kubelet-config\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.856918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-etc-kubernetes\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.856918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cqw\" (UniqueName: \"kubernetes.io/projected/af3193d5-30ab-4869-8ec9-ad6af393c727-kube-api-access-59cqw\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.856999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-cni-multus\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.856999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-kubelet\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.856999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.856999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.856991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-netns\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.857133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-hostroot\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.857133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-cnibin\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.857133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.857133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.857133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-cni-binary-copy\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.857133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-daemon-config\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.857133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-multus-certs\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.857133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/646110b4-56d3-44b3-b75c-3fb2883b274b-dbus\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.857364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-cni-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.857364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5vb6\" (UniqueName: \"kubernetes.io/projected/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-kube-api-access-b5vb6\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.857364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.857364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-system-cni-dir\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.857364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-k8s-cni-cncf-io\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.857364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.857261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-conf-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.858199 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.858183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:26.860312 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.860297    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 12:30:26.860452 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.860433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:26.860518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.860476    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zgr22\""
Sep 10 12:30:26.860518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.860474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 12:30:26.862487 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.862470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:26.862568 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.862546    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:26.862918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.862903    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 12:30:26.863023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.863007    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 12:30:26.863071 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.863029    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 12:30:26.863071 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.863062    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7t76v\""
Sep 10 12:30:26.864681 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.864666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:26.864760 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.864718    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:26.865521 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.865505    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 12:30:26.866845 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.866830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:26.868984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.868967    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:30:26.869231 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.869214    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 12:30:26.869302 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.869214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.869302 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.869256    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bjsg6\""
Sep 10 12:30:26.869398 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.869327    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 12:30:26.871558 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.871545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:26.871870 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.871852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 12:30:26.872263 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.872245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 12:30:26.872409 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.872394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 12:30:26.872588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.872563    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 12:30:26.872701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.872664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 12:30:26.875547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.873015    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qr8pc\""
Sep 10 12:30:26.875547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.874195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.875547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.874940    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 12:30:26.875547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.875153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 12:30:26.875547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.875246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 12:30:26.875547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.875485    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 12:30:26.875935 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.875686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h5g56\""
Sep 10 12:30:26.876919 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.876732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:30:26.876919 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.876746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ll9dp\""
Sep 10 12:30:26.877105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.877055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:26.881288 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.881271    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 12:30:26.881370 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.881347    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 12:30:26.881422 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.881381    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 12:30:26.881468 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.881419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k5h6l\""
Sep 10 12:30:26.883213 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.883197    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:26.894139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.894125    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pspv4"
Sep 10 12:30:26.904125 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.904106    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pspv4"
Sep 10 12:30:26.954202 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.954187    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 12:30:26.957772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-slash\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.957865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-systemd\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.957865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-cni-multus\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.957865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-var-lib-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.958027 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-cni-multus\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958027 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.958027 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64sj\" (UniqueName: \"kubernetes.io/projected/fd93453e-1cad-4867-8586-95ed7eba199c-kube-api-access-p64sj\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.958027 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-tmp-dir\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:26.958027 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.957995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxsrv\" (UniqueName: \"kubernetes.io/projected/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-kube-api-access-qxsrv\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-netns\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-hostroot\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-modprobe-d\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd93453e-1cad-4867-8586-95ed7eba199c-ovn-node-metrics-cert\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-ovnkube-script-lib\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-hostroot\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.958254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-netns\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-cni-binary-copy\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kc7d\" (UniqueName: \"kubernetes.io/projected/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-kube-api-access-4kc7d\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-cni-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f579811-e1ef-4422-b6cc-d78f644e771f-konnectivity-ca\") pod \"konnectivity-agent-6zsm8\" (UID: \"8f579811-e1ef-4422-b6cc-d78f644e771f\") " pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-host\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4t4r\" (UniqueName: \"kubernetes.io/projected/2eb054c1-3406-4776-b4b6-b44b6d1f9849-kube-api-access-k4t4r\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-systemd-units\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-k8s-cni-cncf-io\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.958508    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-conf-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-serviceca\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-systemd\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:26.958618    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret podName:646110b4-56d3-44b3-b75c-3fb2883b274b nodeName:}" failed. No retries permitted until 2026-09-10 12:30:27.458561296 +0000 UTC m=+1.980730329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret") pod "global-pull-secret-syncer-n4575" (UID: "646110b4-56d3-44b3-b75c-3fb2883b274b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-cni-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.958804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-k8s-cni-cncf-io\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-sys\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-cni-bin\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-cni-binary-copy\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-conf-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-ovnkube-config\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-cni-bin\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-run\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.958995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-cni-bin\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-socket-dir-parent\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-sys-fs\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-etc-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-socket-dir-parent\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-etc-kubernetes\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.959572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-etc-kubernetes\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-registration-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-node-log\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-env-overrides\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cqw\" (UniqueName: \"kubernetes.io/projected/af3193d5-30ab-4869-8ec9-ad6af393c727-kube-api-access-59cqw\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af3193d5-30ab-4869-8ec9-ad6af393c727-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-kubelet\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-socket-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-device-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-cni-netd\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-var-lib-kubelet\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-cnibin\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-daemon-config\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-cnibin\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.960354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-multus-certs\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/646110b4-56d3-44b3-b75c-3fb2883b274b-dbus\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-host-run-multus-certs\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8qh\" (UniqueName: \"kubernetes.io/projected/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-kube-api-access-7x8qh\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/646110b4-56d3-44b3-b75c-3fb2883b274b-dbus\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysconfig\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.959972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-hosts-file\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5vb6\" (UniqueName: \"kubernetes.io/projected/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-kube-api-access-b5vb6\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-system-cni-dir\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-lib-modules\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-tuned\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-run-netns\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-ovn\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-system-cni-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-cnibin\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af3193d5-30ab-4869-8ec9-ad6af393c727-system-cni-dir\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.960834 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-iptables-alerter-script\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysctl-d\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-log-socket\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-cnibin\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/646110b4-56d3-44b3-b75c-3fb2883b274b-kubelet-config\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-multus-daemon-config\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-host\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-system-cni-dir\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/646110b4-56d3-44b3-b75c-3fb2883b274b-kubelet-config\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-host-slash\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-kubernetes\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysctl-conf\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f478430-33d6-4e48-a8b3-c85a60ed9088-tmp\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-kubelet\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f579811-e1ef-4422-b6cc-d78f644e771f-agent-certs\") pod \"konnectivity-agent-6zsm8\" (UID: \"8f579811-e1ef-4422-b6cc-d78f644e771f\") " pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg87l\" (UniqueName: \"kubernetes.io/projected/ce6aa60f-baad-4357-9ea0-08c64e91d19d-kube-api-access-xg87l\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-var-lib-kubelet\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.961298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxft\" (UniqueName: \"kubernetes.io/projected/4f478430-33d6-4e48-a8b3-c85a60ed9088-kube-api-access-5wxft\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:26.961804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.960473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:26.968971 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.968952    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 12:30:26.972077 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.972056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cqw\" (UniqueName: \"kubernetes.io/projected/af3193d5-30ab-4869-8ec9-ad6af393c727-kube-api-access-59cqw\") pod \"multus-additional-cni-plugins-fwl5z\" (UID: \"af3193d5-30ab-4869-8ec9-ad6af393c727\") " pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:26.972199 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.972070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5vb6\" (UniqueName: \"kubernetes.io/projected/f150d7b0-d033-4c53-95a2-e9c4a1b032f4-kube-api-access-b5vb6\") pod \"multus-lfc2h\" (UID: \"f150d7b0-d033-4c53-95a2-e9c4a1b032f4\") " pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:26.995491 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:26.995471    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:27.022159 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.022115    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 12:25:25 +0000 UTC" deadline="2028-06-21 09:42:14.314764174 +0000 UTC"
Sep 10 12:30:27.022159 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.022158    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15597h11m47.292608612s"
Sep 10 12:30:27.061633 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8qh\" (UniqueName: \"kubernetes.io/projected/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-kube-api-access-7x8qh\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:27.061749 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.061749 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysconfig\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.061749 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-hosts-file\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:27.061749 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-lib-modules\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.061749 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-tuned\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.061749 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-run-netns\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.061749 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-ovn\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-hosts-file\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysconfig\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-run-netns\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-iptables-alerter-script\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysctl-d\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-log-socket\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-host\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-log-socket\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-ovn\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-lib-modules\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-host-slash\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061956    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-host-slash\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-host\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysctl-d\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.061987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-kubernetes\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysctl-conf\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-kubernetes\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f478430-33d6-4e48-a8b3-c85a60ed9088-tmp\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-kubelet\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f579811-e1ef-4422-b6cc-d78f644e771f-agent-certs\") pod \"konnectivity-agent-6zsm8\" (UID: \"8f579811-e1ef-4422-b6cc-d78f644e771f\") " pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-sysctl-conf\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg87l\" (UniqueName: \"kubernetes.io/projected/ce6aa60f-baad-4357-9ea0-08c64e91d19d-kube-api-access-xg87l\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-var-lib-kubelet\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxft\" (UniqueName: \"kubernetes.io/projected/4f478430-33d6-4e48-a8b3-c85a60ed9088-kube-api-access-5wxft\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-slash\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-systemd\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-iptables-alerter-script\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-var-lib-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-var-lib-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-kubelet\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64sj\" (UniqueName: \"kubernetes.io/projected/fd93453e-1cad-4867-8586-95ed7eba199c-kube-api-access-p64sj\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.062771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-slash\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-tmp-dir\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.062474    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxsrv\" (UniqueName: \"kubernetes.io/projected/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-kube-api-access-qxsrv\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.062542    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs podName:2eb054c1-3406-4776-b4b6-b44b6d1f9849 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:27.562523228 +0000 UTC m=+2.084692265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs") pod "network-metrics-daemon-dj27m" (UID: "2eb054c1-3406-4776-b4b6-b44b6d1f9849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-systemd\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-var-lib-kubelet\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-tmp-dir\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-modprobe-d\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-modprobe-d\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd93453e-1cad-4867-8586-95ed7eba199c-ovn-node-metrics-cert\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-ovnkube-script-lib\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kc7d\" (UniqueName: \"kubernetes.io/projected/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-kube-api-access-4kc7d\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f579811-e1ef-4422-b6cc-d78f644e771f-konnectivity-ca\") pod \"konnectivity-agent-6zsm8\" (UID: \"8f579811-e1ef-4422-b6cc-d78f644e771f\") " pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:27.063528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-host\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4t4r\" (UniqueName: \"kubernetes.io/projected/2eb054c1-3406-4776-b4b6-b44b6d1f9849-kube-api-access-k4t4r\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.062985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-systemd-units\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-systemd-units\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-serviceca\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-systemd\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-sys\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-cni-bin\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063152    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-host\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-ovnkube-config\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-run\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-sys-fs\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-etc-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-registration-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.064426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-node-log\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-env-overrides\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-socket-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-device-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-cni-netd\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-sys-fs\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-cni-netd\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-ovnkube-script-lib\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-etc-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f579811-e1ef-4422-b6cc-d78f644e771f-konnectivity-ca\") pod \"konnectivity-agent-6zsm8\" (UID: \"8f579811-e1ef-4422-b6cc-d78f644e771f\") " pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-host-cni-bin\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-systemd\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-run\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-registration-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-socket-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.063958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-node-log\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.064003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd93453e-1cad-4867-8586-95ed7eba199c-run-openvswitch\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.064009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce6aa60f-baad-4357-9ea0-08c64e91d19d-device-dir\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.065813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.064045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f478430-33d6-4e48-a8b3-c85a60ed9088-sys\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.065813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.064127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-ovnkube-config\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.064172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd93453e-1cad-4867-8586-95ed7eba199c-env-overrides\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.065813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.064585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-serviceca\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:27.065813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.065096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f579811-e1ef-4422-b6cc-d78f644e771f-agent-certs\") pod \"konnectivity-agent-6zsm8\" (UID: \"8f579811-e1ef-4422-b6cc-d78f644e771f\") " pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:27.065813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.065221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f478430-33d6-4e48-a8b3-c85a60ed9088-etc-tuned\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.065813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.065278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd93453e-1cad-4867-8586-95ed7eba199c-ovn-node-metrics-cert\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.066099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.065990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f478430-33d6-4e48-a8b3-c85a60ed9088-tmp\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.069039 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.069017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8qh\" (UniqueName: \"kubernetes.io/projected/804e18ed-32b5-409e-84d7-aa0e77b5c5bd-kube-api-access-7x8qh\") pod \"iptables-alerter-9pvpv\" (UID: \"804e18ed-32b5-409e-84d7-aa0e77b5c5bd\") " pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:27.072382 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.072356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg87l\" (UniqueName: \"kubernetes.io/projected/ce6aa60f-baad-4357-9ea0-08c64e91d19d-kube-api-access-xg87l\") pod \"aws-ebs-csi-driver-node-dp2rj\" (UID: \"ce6aa60f-baad-4357-9ea0-08c64e91d19d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.074943 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.074922    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:27.075035 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.074948    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:27.075035 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.074966    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hcjvx for pod openshift-network-diagnostics/network-check-target-btxk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:27.075148 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.075040    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx podName:63fb7115-8a68-4592-b052-c9b1507c45ff nodeName:}" failed. No retries permitted until 2026-09-10 12:30:27.575023156 +0000 UTC m=+2.097192207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hcjvx" (UniqueName: "kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx") pod "network-check-target-btxk6" (UID: "63fb7115-8a68-4592-b052-c9b1507c45ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:27.076082 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.076057    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxsrv\" (UniqueName: \"kubernetes.io/projected/72c9ae18-ee7a-4949-b2ae-4de97e29e46a-kube-api-access-qxsrv\") pod \"node-resolver-59cz5\" (UID: \"72c9ae18-ee7a-4949-b2ae-4de97e29e46a\") " pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:27.076277 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.076262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxft\" (UniqueName: \"kubernetes.io/projected/4f478430-33d6-4e48-a8b3-c85a60ed9088-kube-api-access-5wxft\") pod \"tuned-h25qd\" (UID: \"4f478430-33d6-4e48-a8b3-c85a60ed9088\") " pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.076657 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.076642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64sj\" (UniqueName: \"kubernetes.io/projected/fd93453e-1cad-4867-8586-95ed7eba199c-kube-api-access-p64sj\") pod \"ovnkube-node-ftz99\" (UID: \"fd93453e-1cad-4867-8586-95ed7eba199c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.077059 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.077043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4t4r\" (UniqueName: \"kubernetes.io/projected/2eb054c1-3406-4776-b4b6-b44b6d1f9849-kube-api-access-k4t4r\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:27.077059 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.077056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kc7d\" (UniqueName: \"kubernetes.io/projected/2a86ebb5-e828-4641-9ffd-c25ab4039f7e-kube-api-access-4kc7d\") pod \"node-ca-rrqld\" (UID: \"2a86ebb5-e828-4641-9ffd-c25ab4039f7e\") " pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:27.139883 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.139835    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f88650c6069898e4f5d2a7851171af.slice/crio-da7848251b4e1304bf94c99d4573bb4919604ee78725268082f932ba2c44f755 WatchSource:0}: Error finding container da7848251b4e1304bf94c99d4573bb4919604ee78725268082f932ba2c44f755: Status 404 returned error can't find the container with id da7848251b4e1304bf94c99d4573bb4919604ee78725268082f932ba2c44f755
Sep 10 12:30:27.140446 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.140420    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad92d93c55b1bd4bbad87ef6544190d.slice/crio-31863b0d60c4ea007d4588135085ef462f63e5bacc3fa725f1bccb0d8af7004f WatchSource:0}: Error finding container 31863b0d60c4ea007d4588135085ef462f63e5bacc3fa725f1bccb0d8af7004f: Status 404 returned error can't find the container with id 31863b0d60c4ea007d4588135085ef462f63e5bacc3fa725f1bccb0d8af7004f
Sep 10 12:30:27.148124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.148103    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:30:27.190124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.190104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fwl5z"
Sep 10 12:30:27.199439 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.199415    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3193d5_30ab_4869_8ec9_ad6af393c727.slice/crio-871ed59534e8ee4f51cea6d11f74e88d0b45ab61026b3007d81a99451d694453 WatchSource:0}: Error finding container 871ed59534e8ee4f51cea6d11f74e88d0b45ab61026b3007d81a99451d694453: Status 404 returned error can't find the container with id 871ed59534e8ee4f51cea6d11f74e88d0b45ab61026b3007d81a99451d694453
Sep 10 12:30:27.200307 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.200293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lfc2h"
Sep 10 12:30:27.207532 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.207513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf150d7b0_d033_4c53_95a2_e9c4a1b032f4.slice/crio-268bf044fb02ddac7e5924359fa98da0bad8c5b9472e0e85f94b680e2bb79987 WatchSource:0}: Error finding container 268bf044fb02ddac7e5924359fa98da0bad8c5b9472e0e85f94b680e2bb79987: Status 404 returned error can't find the container with id 268bf044fb02ddac7e5924359fa98da0bad8c5b9472e0e85f94b680e2bb79987
Sep 10 12:30:27.227215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.227199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:27.234154 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.234137    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f579811_e1ef_4422_b6cc_d78f644e771f.slice/crio-d44c9f229ec258b6b63357e2cdcb921f24504188ac41b1cf1c8f8f55ab9604d0 WatchSource:0}: Error finding container d44c9f229ec258b6b63357e2cdcb921f24504188ac41b1cf1c8f8f55ab9604d0: Status 404 returned error can't find the container with id d44c9f229ec258b6b63357e2cdcb921f24504188ac41b1cf1c8f8f55ab9604d0
Sep 10 12:30:27.239057 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.239037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rrqld"
Sep 10 12:30:27.246670 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.246514    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a86ebb5_e828_4641_9ffd_c25ab4039f7e.slice/crio-3f39a9814975370fbbd19e8c7ef15b89cfb22500ecce261a4f121aa5917a52c9 WatchSource:0}: Error finding container 3f39a9814975370fbbd19e8c7ef15b89cfb22500ecce261a4f121aa5917a52c9: Status 404 returned error can't find the container with id 3f39a9814975370fbbd19e8c7ef15b89cfb22500ecce261a4f121aa5917a52c9
Sep 10 12:30:27.256549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.256535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9pvpv"
Sep 10 12:30:27.264124 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.264098    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804e18ed_32b5_409e_84d7_aa0e77b5c5bd.slice/crio-0b94d96e164ccdc1ba146c712aefa01a21bbc88b5c125f9df26e6e98d17ea9c8 WatchSource:0}: Error finding container 0b94d96e164ccdc1ba146c712aefa01a21bbc88b5c125f9df26e6e98d17ea9c8: Status 404 returned error can't find the container with id 0b94d96e164ccdc1ba146c712aefa01a21bbc88b5c125f9df26e6e98d17ea9c8
Sep 10 12:30:27.271048 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.271032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:27.278326 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.278310    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd93453e_1cad_4867_8586_95ed7eba199c.slice/crio-57599343b89842c2d477d8ec6b4e8c8f7c83ca07a804cae013df42c02d437ae3 WatchSource:0}: Error finding container 57599343b89842c2d477d8ec6b4e8c8f7c83ca07a804cae013df42c02d437ae3: Status 404 returned error can't find the container with id 57599343b89842c2d477d8ec6b4e8c8f7c83ca07a804cae013df42c02d437ae3
Sep 10 12:30:27.298671 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.298652    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj"
Sep 10 12:30:27.306936 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.306915    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6aa60f_baad_4357_9ea0_08c64e91d19d.slice/crio-240a771fc248847a112cbe90016f20d0f27029ac9746caa4bd1a8710b4cb09bc WatchSource:0}: Error finding container 240a771fc248847a112cbe90016f20d0f27029ac9746caa4bd1a8710b4cb09bc: Status 404 returned error can't find the container with id 240a771fc248847a112cbe90016f20d0f27029ac9746caa4bd1a8710b4cb09bc
Sep 10 12:30:27.313760 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.313744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h25qd"
Sep 10 12:30:27.318418 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.318402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-59cz5"
Sep 10 12:30:27.321143 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.321120    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f478430_33d6_4e48_a8b3_c85a60ed9088.slice/crio-40693691ba5a935d038f5ebcb84bc927ce73f0d4d8e58525884d69dfeb7e36be WatchSource:0}: Error finding container 40693691ba5a935d038f5ebcb84bc927ce73f0d4d8e58525884d69dfeb7e36be: Status 404 returned error can't find the container with id 40693691ba5a935d038f5ebcb84bc927ce73f0d4d8e58525884d69dfeb7e36be
Sep 10 12:30:27.326056 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:30:27.326039    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c9ae18_ee7a_4949_b2ae_4de97e29e46a.slice/crio-3e3e17ce9e4c43d49a41e5ecae326d18651a9490f588998cc00b5ee59a354abe WatchSource:0}: Error finding container 3e3e17ce9e4c43d49a41e5ecae326d18651a9490f588998cc00b5ee59a354abe: Status 404 returned error can't find the container with id 3e3e17ce9e4c43d49a41e5ecae326d18651a9490f588998cc00b5ee59a354abe
Sep 10 12:30:27.327484 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.327461    2577 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-59cz5_openshift-dns(72c9ae18-ee7a-4949-b2ae-4de97e29e46a): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 10 12:30:27.327527 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.327507    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-59cz5" podUID="72c9ae18-ee7a-4949-b2ae-4de97e29e46a"
Sep 10 12:30:27.439918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.439852    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:27.466086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.466063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:27.466192 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.466180    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:27.466256 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.466222    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret podName:646110b4-56d3-44b3-b75c-3fb2883b274b nodeName:}" failed. No retries permitted until 2026-09-10 12:30:28.466210954 +0000 UTC m=+2.988379984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret") pod "global-pull-secret-syncer-n4575" (UID: "646110b4-56d3-44b3-b75c-3fb2883b274b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:27.566700 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.566673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:27.566836 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.566767    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:27.566836 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.566816    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs podName:2eb054c1-3406-4776-b4b6-b44b6d1f9849 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:28.56680467 +0000 UTC m=+3.088973700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs") pod "network-metrics-daemon-dj27m" (UID: "2eb054c1-3406-4776-b4b6-b44b6d1f9849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:27.667009 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.666976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:27.667184 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.667161    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:27.667239 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.667196    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:27.667239 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.667210    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hcjvx for pod openshift-network-diagnostics/network-check-target-btxk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:27.667345 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.667281    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx podName:63fb7115-8a68-4592-b052-c9b1507c45ff nodeName:}" failed. No retries permitted until 2026-09-10 12:30:28.66725825 +0000 UTC m=+3.189427286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hcjvx" (UniqueName: "kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx") pod "network-check-target-btxk6" (UID: "63fb7115-8a68-4592-b052-c9b1507c45ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:27.908000 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.907959    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 12:25:26 +0000 UTC" deadline="2028-05-24 11:24:49.428093804 +0000 UTC"
Sep 10 12:30:27.908000 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.907999    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14926h54m21.52010002s"
Sep 10 12:30:27.996521 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:27.996490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:27.996698 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:27.996637    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:28.019948 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.019886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-59cz5" event={"ID":"72c9ae18-ee7a-4949-b2ae-4de97e29e46a","Type":"ContainerStarted","Data":"3e3e17ce9e4c43d49a41e5ecae326d18651a9490f588998cc00b5ee59a354abe"}
Sep 10 12:30:28.023241 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.023007    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40bc1b9c6b608369978e9f165c5e22d7a75fd91b3480cba60cd275228d721616\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-59cz5" podUID="72c9ae18-ee7a-4949-b2ae-4de97e29e46a"
Sep 10 12:30:28.038022 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.036949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h25qd" event={"ID":"4f478430-33d6-4e48-a8b3-c85a60ed9088","Type":"ContainerStarted","Data":"40693691ba5a935d038f5ebcb84bc927ce73f0d4d8e58525884d69dfeb7e36be"}
Sep 10 12:30:28.053123 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.053041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj" event={"ID":"ce6aa60f-baad-4357-9ea0-08c64e91d19d","Type":"ContainerStarted","Data":"240a771fc248847a112cbe90016f20d0f27029ac9746caa4bd1a8710b4cb09bc"}
Sep 10 12:30:28.061030 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.060947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"57599343b89842c2d477d8ec6b4e8c8f7c83ca07a804cae013df42c02d437ae3"}
Sep 10 12:30:28.075409 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.075381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" event={"ID":"af3193d5-30ab-4869-8ec9-ad6af393c727","Type":"ContainerStarted","Data":"871ed59534e8ee4f51cea6d11f74e88d0b45ab61026b3007d81a99451d694453"}
Sep 10 12:30:28.086170 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.086144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" event={"ID":"f2f88650c6069898e4f5d2a7851171af","Type":"ContainerStarted","Data":"da7848251b4e1304bf94c99d4573bb4919604ee78725268082f932ba2c44f755"}
Sep 10 12:30:28.107269 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.107241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" event={"ID":"9ad92d93c55b1bd4bbad87ef6544190d","Type":"ContainerStarted","Data":"31863b0d60c4ea007d4588135085ef462f63e5bacc3fa725f1bccb0d8af7004f"}
Sep 10 12:30:28.131416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.131385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9pvpv" event={"ID":"804e18ed-32b5-409e-84d7-aa0e77b5c5bd","Type":"ContainerStarted","Data":"0b94d96e164ccdc1ba146c712aefa01a21bbc88b5c125f9df26e6e98d17ea9c8"}
Sep 10 12:30:28.143585 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.143550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rrqld" event={"ID":"2a86ebb5-e828-4641-9ffd-c25ab4039f7e","Type":"ContainerStarted","Data":"3f39a9814975370fbbd19e8c7ef15b89cfb22500ecce261a4f121aa5917a52c9"}
Sep 10 12:30:28.167698 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.167614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6zsm8" event={"ID":"8f579811-e1ef-4422-b6cc-d78f644e771f","Type":"ContainerStarted","Data":"d44c9f229ec258b6b63357e2cdcb921f24504188ac41b1cf1c8f8f55ab9604d0"}
Sep 10 12:30:28.172428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.172400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lfc2h" event={"ID":"f150d7b0-d033-4c53-95a2-e9c4a1b032f4","Type":"ContainerStarted","Data":"268bf044fb02ddac7e5924359fa98da0bad8c5b9472e0e85f94b680e2bb79987"}
Sep 10 12:30:28.473901 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.473792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:28.474057 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.473939    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:28.474057 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.474002    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret podName:646110b4-56d3-44b3-b75c-3fb2883b274b nodeName:}" failed. No retries permitted until 2026-09-10 12:30:30.473982606 +0000 UTC m=+4.996151636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret") pod "global-pull-secret-syncer-n4575" (UID: "646110b4-56d3-44b3-b75c-3fb2883b274b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:28.574692 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.574658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:28.574864 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.574819    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:28.574936 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.574887    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs podName:2eb054c1-3406-4776-b4b6-b44b6d1f9849 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:30.574867351 +0000 UTC m=+5.097036390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs") pod "network-metrics-daemon-dj27m" (UID: "2eb054c1-3406-4776-b4b6-b44b6d1f9849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:28.675501 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.675465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:28.675719 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.675696    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:28.675832 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.675727    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:28.675832 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.675741    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hcjvx for pod openshift-network-diagnostics/network-check-target-btxk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:28.675832 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.675802    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx podName:63fb7115-8a68-4592-b052-c9b1507c45ff nodeName:}" failed. No retries permitted until 2026-09-10 12:30:30.675781911 +0000 UTC m=+5.197950948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hcjvx" (UniqueName: "kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx") pod "network-check-target-btxk6" (UID: "63fb7115-8a68-4592-b052-c9b1507c45ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:28.908315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.908242    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 12:25:26 +0000 UTC" deadline="2028-03-19 21:10:06.320079758 +0000 UTC"
Sep 10 12:30:28.908315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.908306    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13352h39m37.411777456s"
Sep 10 12:30:28.994368 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.994236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:28.994368 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.994366    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:28.994659 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:28.994441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:28.994659 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:28.994525    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:29.859134 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:29.859100    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:29.996509 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:29.996480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:29.996986 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:29.996622    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:30.493680 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:30.493544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:30.493832 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.493736    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:30.493832 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.493808    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret podName:646110b4-56d3-44b3-b75c-3fb2883b274b nodeName:}" failed. No retries permitted until 2026-09-10 12:30:34.493790361 +0000 UTC m=+9.015959397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret") pod "global-pull-secret-syncer-n4575" (UID: "646110b4-56d3-44b3-b75c-3fb2883b274b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:30.594868 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:30.594832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:30.595065 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.595028    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:30.595162 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.595117    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs podName:2eb054c1-3406-4776-b4b6-b44b6d1f9849 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:34.595097204 +0000 UTC m=+9.117266248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs") pod "network-metrics-daemon-dj27m" (UID: "2eb054c1-3406-4776-b4b6-b44b6d1f9849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:30.696189 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:30.696156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:30.696358 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.696307    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:30.696358 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.696328    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:30.696358 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.696341    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hcjvx for pod openshift-network-diagnostics/network-check-target-btxk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:30.696526 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.696404    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx podName:63fb7115-8a68-4592-b052-c9b1507c45ff nodeName:}" failed. No retries permitted until 2026-09-10 12:30:34.696385606 +0000 UTC m=+9.218554641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hcjvx" (UniqueName: "kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx") pod "network-check-target-btxk6" (UID: "63fb7115-8a68-4592-b052-c9b1507c45ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:30.994427 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:30.994393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:30.994427 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:30.994435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:30.994700 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.994533    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:30.994700 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:30.994673    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:31.994621 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:31.993958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:31.994621 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:31.994077    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:32.994488 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:32.994452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:32.994488 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:32.994474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:32.995039 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:32.994633    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:32.995039 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:32.994755    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:33.994828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:33.994793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:33.995279 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:33.994924    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:34.528006 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:34.527955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:34.528175 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.528114    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:34.528235 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.528177    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret podName:646110b4-56d3-44b3-b75c-3fb2883b274b nodeName:}" failed. No retries permitted until 2026-09-10 12:30:42.528159552 +0000 UTC m=+17.050328585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret") pod "global-pull-secret-syncer-n4575" (UID: "646110b4-56d3-44b3-b75c-3fb2883b274b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:34.629166 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:34.629118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:34.629341 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.629288    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:34.629402 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.629362    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs podName:2eb054c1-3406-4776-b4b6-b44b6d1f9849 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:42.629342358 +0000 UTC m=+17.151511473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs") pod "network-metrics-daemon-dj27m" (UID: "2eb054c1-3406-4776-b4b6-b44b6d1f9849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:34.730229 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:34.730140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:34.730396 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.730263    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:34.730396 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.730277    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:34.730396 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.730286    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hcjvx for pod openshift-network-diagnostics/network-check-target-btxk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:34.730396 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.730348    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx podName:63fb7115-8a68-4592-b052-c9b1507c45ff nodeName:}" failed. No retries permitted until 2026-09-10 12:30:42.73032877 +0000 UTC m=+17.252497803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hcjvx" (UniqueName: "kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx") pod "network-check-target-btxk6" (UID: "63fb7115-8a68-4592-b052-c9b1507c45ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:34.994460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:34.994369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:34.994460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:34.994421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:34.994681 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.994509    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:34.994757 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:34.994672    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:35.995673 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:35.995638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:35.996114 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:35.995764    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:36.994590 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:36.994547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:36.994762 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:36.994608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:36.994762 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:36.994704    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:36.994886 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:36.994831    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:37.994650 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:37.994618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:37.995240 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:37.994730    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:38.994172 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:38.994097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:38.994172 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:38.994137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:38.994411 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:38.994237    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:38.994540 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:38.994500    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:39.994149 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:39.994115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:39.994591 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:39.994236    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:40.994270 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:40.994233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:40.994270 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:40.994256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:40.994763 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:40.994353    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:40.994763 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:40.994502    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:41.994097 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:41.994014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:41.994257 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:41.994175    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:42.584867 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:42.584834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:42.585359 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.584961    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:42.585359 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.585032    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret podName:646110b4-56d3-44b3-b75c-3fb2883b274b nodeName:}" failed. No retries permitted until 2026-09-10 12:30:58.585012635 +0000 UTC m=+33.107181669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret") pod "global-pull-secret-syncer-n4575" (UID: "646110b4-56d3-44b3-b75c-3fb2883b274b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:42.686165 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:42.686120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:42.686334 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.686260    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:42.686334 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.686323    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs podName:2eb054c1-3406-4776-b4b6-b44b6d1f9849 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:58.686305906 +0000 UTC m=+33.208474940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs") pod "network-metrics-daemon-dj27m" (UID: "2eb054c1-3406-4776-b4b6-b44b6d1f9849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:42.786951 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:42.786916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:42.787115 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.787064    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:42.787115 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.787085    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:42.787115 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.787098    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hcjvx for pod openshift-network-diagnostics/network-check-target-btxk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:42.787261 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.787147    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx podName:63fb7115-8a68-4592-b052-c9b1507c45ff nodeName:}" failed. No retries permitted until 2026-09-10 12:30:58.787133369 +0000 UTC m=+33.309302398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hcjvx" (UniqueName: "kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx") pod "network-check-target-btxk6" (UID: "63fb7115-8a68-4592-b052-c9b1507c45ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:42.994293 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:42.994220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:42.994463 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:42.994232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:42.994463 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.994346    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:42.994463 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:42.994443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:43.993927 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:43.993890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:43.994328 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:43.994001    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:44.994355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:44.994318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:44.994750 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:44.994319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:44.994750 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:44.994482    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:44.994750 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:44.994510    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:45.995220 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:45.994997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:45.995673 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:45.995312    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:46.209515 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.209492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lfc2h" event={"ID":"f150d7b0-d033-4c53-95a2-e9c4a1b032f4","Type":"ContainerStarted","Data":"0c70d144f0dba8e7b292d7ada35b52070dba5103ea797c0b9b9cef4033d89106"}
Sep 10 12:30:46.211031 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.211011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h25qd" event={"ID":"4f478430-33d6-4e48-a8b3-c85a60ed9088","Type":"ContainerStarted","Data":"e834a92efb0b6b16c4ec51a5e26bd79e7268e6b8b0f412da16daa7b3172be5af"}
Sep 10 12:30:46.213370 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.213348    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:30:46.213738 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.213685    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd93453e-1cad-4867-8586-95ed7eba199c" containerID="bb0e006430c7abaf847df24747dff828b3aabef0aedd2793fb8f453378f6f749" exitCode=1
Sep 10 12:30:46.213824 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.213770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"0bb7d9c4b190779b15100d1358a17a0a281ef6cb529e5101275ecd0a17f1930a"}
Sep 10 12:30:46.213877 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.213842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"97b89ffa75fab80b679185f5a2e7abb7fe819261027e0a11781ef3950fc5dd8d"}
Sep 10 12:30:46.213927 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.213893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerDied","Data":"bb0e006430c7abaf847df24747dff828b3aabef0aedd2793fb8f453378f6f749"}
Sep 10 12:30:46.213972 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.213943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"215329f5e32761c16b34b11253a8d10f2bac00d166e6f7dcef336e70cadb3aff"}
Sep 10 12:30:46.216916 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.216891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" event={"ID":"9ad92d93c55b1bd4bbad87ef6544190d","Type":"ContainerStarted","Data":"cb38d518f7f8868e641d6bdbcb96d40edc66ba1634833436f0f99c0badb5c9e9"}
Sep 10 12:30:46.232481 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.232426    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lfc2h" podStartSLOduration=1.381138096 podStartE2EDuration="20.232406158s" totalImagesPullingTime="18.851268062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:46.23046401 +0000 UTC m=+20.752633062" watchObservedRunningTime="2026-09-10 12:30:46.232406158 +0000 UTC m=+20.754575235"
Sep 10 12:30:46.250094 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.249843    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h25qd" podStartSLOduration=1.959563541 podStartE2EDuration="20.249824586s" totalImagesPullingTime="18.290261045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:46.249132895 +0000 UTC m=+20.771301970" watchObservedRunningTime="2026-09-10 12:30:46.249824586 +0000 UTC m=+20.771993655"
Sep 10 12:30:46.268541 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.268453    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-169.ec2.internal" podStartSLOduration=20.268431371 podStartE2EDuration="20.268431371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:46.267480593 +0000 UTC m=+20.789649649" watchObservedRunningTime="2026-09-10 12:30:46.268431371 +0000 UTC m=+20.790600424"
Sep 10 12:30:46.994835 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.994665    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:46.994972 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:46.994665    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:46.994972 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:46.994925    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:46.995065 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:46.994973    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:47.220233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.220192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj" event={"ID":"ce6aa60f-baad-4357-9ea0-08c64e91d19d","Type":"ContainerStarted","Data":"4dc59f4a71c1e9cbcbb43cb678875592a37ad273facfc64291da4e93f989c4ed"}
Sep 10 12:30:47.223290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.223268    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:30:47.223802 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.223773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"7c5f8d74cc2a8381fd27245c475b5541200d24502a6863aee75b5b51dbe4f1c1"}
Sep 10 12:30:47.223888 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.223850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"d12f23ee80d5d65c98c1570522be4c9739cce77528d38e4cd8cb07fa61589e29"}
Sep 10 12:30:47.225576 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.225549    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3193d5-30ab-4869-8ec9-ad6af393c727" containerID="a0d82465008a7651904c1aa669b545a42c22ea715d9807274b7759ed34602143" exitCode=0
Sep 10 12:30:47.225678 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.225636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" event={"ID":"af3193d5-30ab-4869-8ec9-ad6af393c727","Type":"ContainerDied","Data":"a0d82465008a7651904c1aa669b545a42c22ea715d9807274b7759ed34602143"}
Sep 10 12:30:47.227360 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.227339    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2f88650c6069898e4f5d2a7851171af" containerID="0cbbf859df387770342c6a830a46a8d0cf8927749714a635618983ef73451825" exitCode=0
Sep 10 12:30:47.227450 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.227421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" event={"ID":"f2f88650c6069898e4f5d2a7851171af","Type":"ContainerDied","Data":"0cbbf859df387770342c6a830a46a8d0cf8927749714a635618983ef73451825"}
Sep 10 12:30:47.229512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.229462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9pvpv" event={"ID":"804e18ed-32b5-409e-84d7-aa0e77b5c5bd","Type":"ContainerStarted","Data":"b1244dd311a96e6d2293bbc83196921385eaf708edc7dda6c6e09d516afab5b7"}
Sep 10 12:30:47.232820 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.232788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rrqld" event={"ID":"2a86ebb5-e828-4641-9ffd-c25ab4039f7e","Type":"ContainerStarted","Data":"ce1c02c0a2975e11ab99b6286f8e9ae3076aeeda227bb064aae344e6666c0e61"}
Sep 10 12:30:47.235156 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.235132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6zsm8" event={"ID":"8f579811-e1ef-4422-b6cc-d78f644e771f","Type":"ContainerStarted","Data":"be63c96d5253c8481d234402074ed5dcc8729091f3de7fe2615fe5d7c8382bf5"}
Sep 10 12:30:47.236851 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.236825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-59cz5" event={"ID":"72c9ae18-ee7a-4949-b2ae-4de97e29e46a","Type":"ContainerStarted","Data":"c165b328f2ad9e49cbd5f6729a4b590b568a462522ca9cba26159743643adc1e"}
Sep 10 12:30:47.264159 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.264117    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-59cz5" podStartSLOduration=2.816784822 podStartE2EDuration="21.264103417s" totalImagesPullingTime="18.447318595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-10 12:30:47.263708541 +0000 UTC m=+21.785877625" watchObservedRunningTime="2026-09-10 12:30:47.264103417 +0000 UTC m=+21.786272469"
Sep 10 12:30:47.293746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.293702    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6zsm8" podStartSLOduration=2.916610767 podStartE2EDuration="21.293686455s" totalImagesPullingTime="18.377075688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:47.29321275 +0000 UTC m=+21.815381803" watchObservedRunningTime="2026-09-10 12:30:47.293686455 +0000 UTC m=+21.815855544"
Sep 10 12:30:47.308941 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.308895    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rrqld" podStartSLOduration=2.989681266 podStartE2EDuration="21.308876779s" totalImagesPullingTime="18.319195513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:47.308369678 +0000 UTC m=+21.830538730" watchObservedRunningTime="2026-09-10 12:30:47.308876779 +0000 UTC m=+21.831045835"
Sep 10 12:30:47.321941 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.321905    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9pvpv" podStartSLOduration=2.97371557 podStartE2EDuration="21.32188557s" totalImagesPullingTime="18.34817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:47.321877075 +0000 UTC m=+21.844046141" watchObservedRunningTime="2026-09-10 12:30:47.32188557 +0000 UTC m=+21.844054609"
Sep 10 12:30:47.464804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.464759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:47.543424 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.543390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:47.643256 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.643213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:47.895073 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.895051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:47.935569 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.935542    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 12:30:47.961746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.961719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:47.993853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:47.993831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:47.993954 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:47.993937    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:48.058943 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.058915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:48.066918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.066853    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T12:30:47.935566429Z","UUID":"bda6de5f-2025-49dc-b74c-5d09333f9065","Handler":null,"Name":"","Endpoint":""}
Sep 10 12:30:48.068159 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.068144    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 12:30:48.068244 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.068169    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 12:30:48.145531 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.145448    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:48.241189 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.241156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj" event={"ID":"ce6aa60f-baad-4357-9ea0-08c64e91d19d","Type":"ContainerStarted","Data":"871f6c529c451eea0a5b75843a6460fd500c66af6c0a47ce94aa32ce36933824"}
Sep 10 12:30:48.243024 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.242989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" event={"ID":"f2f88650c6069898e4f5d2a7851171af","Type":"ContainerStarted","Data":"6cdf3508a4c152a119f00b4d10a2068348247e9418ab0c897b63ecfde107c51b"}
Sep 10 12:30:48.262292 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.262240    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-169.ec2.internal" podStartSLOduration=22.262220563 podStartE2EDuration="22.262220563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:48.26158942 +0000 UTC m=+22.783758471" watchObservedRunningTime="2026-09-10 12:30:48.262220563 +0000 UTC m=+22.784389628"
Sep 10 12:30:48.977943 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.977773    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:48.994048 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.994019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:48.994185 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:48.994054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:48.994185 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:48.994129    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:48.994274 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:48.994186    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:49.077579 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:49.077547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:49.250108 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:49.250087    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:30:49.250582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:49.250555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"f4564e0b8771517204ee5697ca391c1b396b68661910edfab65b0a83983c4dc7"}
Sep 10 12:30:49.800378 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:49.800336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:49.901095 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:49.901065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:49.994588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:49.994548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:49.994798 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:49.994694    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:50.107435 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:50.107340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:50.170578 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:50.170544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:50.254728 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:50.254684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj" event={"ID":"ce6aa60f-baad-4357-9ea0-08c64e91d19d","Type":"ContainerStarted","Data":"da0b08175a34702e8619d1c424a9426504329b26654b0df670258f64ca7db921"}
Sep 10 12:30:50.274671 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:50.274619    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp2rj" podStartSLOduration=3.442278267 podStartE2EDuration="24.274588432s" totalImagesPullingTime="20.832310165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:50.274365467 +0000 UTC m=+24.796534520" watchObservedRunningTime="2026-09-10 12:30:50.274588432 +0000 UTC m=+24.796757487"
Sep 10 12:30:50.994883 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:50.994790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:50.995038 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:50.994790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:50.995038 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:50.994924    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:50.995146 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:50.995034    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:51.261581 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:51.261515    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:30:51.261988 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:51.261936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"327de5aff06f0fc8b271847be480510d8b82f91a2670f43419c34bbfb8a4d92c"}
Sep 10 12:30:51.262541 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:51.262519    2577 scope.go:117] "RemoveContainer" containerID="bb0e006430c7abaf847df24747dff828b3aabef0aedd2793fb8f453378f6f749"
Sep 10 12:30:51.924203 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:51.923999    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:51.924647 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:51.924625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:51.994563 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:51.994536    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:51.994714 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:51.994667    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:52.265899 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.265807    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:52.266391 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.266367    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6zsm8"
Sep 10 12:30:52.330690 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.330666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:52.509058 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.509032    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dj27m"]
Sep 10 12:30:52.509173 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.509161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:52.509303 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:52.509273    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:52.509953 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.509927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-btxk6"]
Sep 10 12:30:52.510034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.510021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:52.510148 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:52.510130    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:52.522751 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.522726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n4575"]
Sep 10 12:30:52.522863 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:52.522843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:52.522933 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:52.522916    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:53.135258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.135188    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:53.197094 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.197064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:53.271512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.271491    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:30:53.272241 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.271918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" event={"ID":"fd93453e-1cad-4867-8586-95ed7eba199c","Type":"ContainerStarted","Data":"80a6d3bc6d7a34be01381e79e803148f46d03111a079cac40f0b2a9336884da3"}
Sep 10 12:30:53.272241 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.271992    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 12:30:53.272241 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.272209    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:53.272384 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.272284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:53.273951 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.273930    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3193d5-30ab-4869-8ec9-ad6af393c727" containerID="831f4fc6885d810681be1b2af3720d1b0db9b30aa4ce5ec9f56749100a2111c5" exitCode=0
Sep 10 12:30:53.274053 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.274024    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" event={"ID":"af3193d5-30ab-4869-8ec9-ad6af393c727","Type":"ContainerDied","Data":"831f4fc6885d810681be1b2af3720d1b0db9b30aa4ce5ec9f56749100a2111c5"}
Sep 10 12:30:53.287406 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.287386    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:53.287584 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.287571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:53.303745 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.303695    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" podStartSLOduration=8.949423384 podStartE2EDuration="27.303683094s" totalImagesPullingTime="18.35425971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:53.302686984 +0000 UTC m=+27.824856036" watchObservedRunningTime="2026-09-10 12:30:53.303683094 +0000 UTC m=+27.825852124"
Sep 10 12:30:53.994073 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.994044    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:53.994246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.994051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:53.994246 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:53.994149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:53.994340 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:53.994244    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:53.994340 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:53.994048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:53.994412 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:53.994357    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:54.131891 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:54.131860    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:30:55.280381 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:55.280193    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3193d5-30ab-4869-8ec9-ad6af393c727" containerID="6c9d0860e5db5b0c990f096816c6c490ac48a42da01d37652d4dab0b7f4807c5" exitCode=0
Sep 10 12:30:55.280899 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:55.280269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" event={"ID":"af3193d5-30ab-4869-8ec9-ad6af393c727","Type":"ContainerDied","Data":"6c9d0860e5db5b0c990f096816c6c490ac48a42da01d37652d4dab0b7f4807c5"}
Sep 10 12:30:55.294777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:55.294745    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99" podUID="fd93453e-1cad-4867-8586-95ed7eba199c" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 12:30:55.994911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:55.994881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:55.995063 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:55.994976    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:55.995105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:55.995079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:55.995143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:55.995129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:55.995218 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:55.995201    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:55.995280 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:55.995266    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:57.287056 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:57.287021    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3193d5-30ab-4869-8ec9-ad6af393c727" containerID="399328936db44146b9b9f00737a711480183f13f058bbdff2591717190c8ec58" exitCode=0
Sep 10 12:30:57.287559 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:57.287079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" event={"ID":"af3193d5-30ab-4869-8ec9-ad6af393c727","Type":"ContainerDied","Data":"399328936db44146b9b9f00737a711480183f13f058bbdff2591717190c8ec58"}
Sep 10 12:30:57.994711 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:57.994680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:57.994878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:57.994692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:57.994878 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:57.994790    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-btxk6" podUID="63fb7115-8a68-4592-b052-c9b1507c45ff"
Sep 10 12:30:57.994878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:57.994680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:57.994878 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:57.994867    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj27m" podUID="2eb054c1-3406-4776-b4b6-b44b6d1f9849"
Sep 10 12:30:57.995012 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:57.994972    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n4575" podUID="646110b4-56d3-44b3-b75c-3fb2883b274b"
Sep 10 12:30:58.601172 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.601141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:58.601561 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:58.601258    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:58.601561 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:58.601318    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret podName:646110b4-56d3-44b3-b75c-3fb2883b274b nodeName:}" failed. No retries permitted until 2026-09-10 12:31:30.601304355 +0000 UTC m=+65.123473528 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret") pod "global-pull-secret-syncer-n4575" (UID: "646110b4-56d3-44b3-b75c-3fb2883b274b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:58.701775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.701742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:58.701921 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:58.701875    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:58.701986 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:58.701953    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs podName:2eb054c1-3406-4776-b4b6-b44b6d1f9849 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:30.701933287 +0000 UTC m=+65.224102321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs") pod "network-metrics-daemon-dj27m" (UID: "2eb054c1-3406-4776-b4b6-b44b6d1f9849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:58.709449 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.709429    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-169.ec2.internal" event="NodeReady"
Sep 10 12:30:58.709547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.709542    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 12:30:58.765735 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.765704    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xthlv"]
Sep 10 12:30:58.778406 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.778384    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jf9x8"]
Sep 10 12:30:58.778533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.778505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xthlv"
Sep 10 12:30:58.781315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.781294    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 12:30:58.784854 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.784746    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 12:30:58.785038 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.785010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zj5jx\""
Sep 10 12:30:58.785147 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.785129    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 12:30:58.790862 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.790831    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xthlv"]
Sep 10 12:30:58.790862 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.790865    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jf9x8"]
Sep 10 12:30:58.791025 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.790959    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:58.794051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.794029    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 12:30:58.794051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.794045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgmkk\""
Sep 10 12:30:58.794214 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.794121    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 12:30:58.802216 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.802195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:58.802319 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:58.802303    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:58.802417 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:58.802324    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:58.802417 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:58.802333    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hcjvx for pod openshift-network-diagnostics/network-check-target-btxk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:58.802417 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:58.802382    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx podName:63fb7115-8a68-4592-b052-c9b1507c45ff nodeName:}" failed. No retries permitted until 2026-09-10 12:31:30.80236585 +0000 UTC m=+65.324534884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hcjvx" (UniqueName: "kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx") pod "network-check-target-btxk6" (UID: "63fb7115-8a68-4592-b052-c9b1507c45ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:58.903112 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.903024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2z9j\" (UniqueName: \"kubernetes.io/projected/ee744d16-fa03-4d53-98fe-cb6fea265c0c-kube-api-access-p2z9j\") pod \"ingress-canary-xthlv\" (UID: \"ee744d16-fa03-4d53-98fe-cb6fea265c0c\") " pod="openshift-ingress-canary/ingress-canary-xthlv"
Sep 10 12:30:58.903112 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.903082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-tmp-dir\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:58.903310 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.903145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqnzz\" (UniqueName: \"kubernetes.io/projected/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-kube-api-access-dqnzz\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:58.903310 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.903212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee744d16-fa03-4d53-98fe-cb6fea265c0c-cert\") pod \"ingress-canary-xthlv\" (UID: \"ee744d16-fa03-4d53-98fe-cb6fea265c0c\") " pod="openshift-ingress-canary/ingress-canary-xthlv"
Sep 10 12:30:58.903310 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.903283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-config-volume\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:58.903414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.903315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:58.913855 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:58.913825    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:30:59.004257 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.004227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2z9j\" (UniqueName: \"kubernetes.io/projected/ee744d16-fa03-4d53-98fe-cb6fea265c0c-kube-api-access-p2z9j\") pod \"ingress-canary-xthlv\" (UID: \"ee744d16-fa03-4d53-98fe-cb6fea265c0c\") " pod="openshift-ingress-canary/ingress-canary-xthlv"
Sep 10 12:30:59.004402 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.004271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-tmp-dir\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:59.004402 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.004293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqnzz\" (UniqueName: \"kubernetes.io/projected/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-kube-api-access-dqnzz\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:59.004402 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.004325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee744d16-fa03-4d53-98fe-cb6fea265c0c-cert\") pod \"ingress-canary-xthlv\" (UID: \"ee744d16-fa03-4d53-98fe-cb6fea265c0c\") " pod="openshift-ingress-canary/ingress-canary-xthlv"
Sep 10 12:30:59.004402 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.004380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-config-volume\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:59.004643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.004413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:59.004643 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:59.004501    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:30:59.004643 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:59.004572    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls podName:9dfc4909-2e6d-4b80-a781-ed7e8fa59616 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:59.504551153 +0000 UTC m=+34.026720187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls") pod "dns-default-jf9x8" (UID: "9dfc4909-2e6d-4b80-a781-ed7e8fa59616") : secret "dns-default-metrics-tls" not found
Sep 10 12:30:59.004975 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.004954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-config-volume\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:59.009312 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.009289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee744d16-fa03-4d53-98fe-cb6fea265c0c-cert\") pod \"ingress-canary-xthlv\" (UID: \"ee744d16-fa03-4d53-98fe-cb6fea265c0c\") " pod="openshift-ingress-canary/ingress-canary-xthlv"
Sep 10 12:30:59.012623 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.012417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-tmp-dir\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:59.013006 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.012984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqnzz\" (UniqueName: \"kubernetes.io/projected/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-kube-api-access-dqnzz\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:59.013473 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.013454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2z9j\" (UniqueName: \"kubernetes.io/projected/ee744d16-fa03-4d53-98fe-cb6fea265c0c-kube-api-access-p2z9j\") pod \"ingress-canary-xthlv\" (UID: \"ee744d16-fa03-4d53-98fe-cb6fea265c0c\") " pod="openshift-ingress-canary/ingress-canary-xthlv"
Sep 10 12:30:59.091571 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.091530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xthlv"
Sep 10 12:30:59.410417 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.408146    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xthlv"]
Sep 10 12:30:59.508684 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.508591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:30:59.508875 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:59.508764    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:30:59.508875 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:30:59.508837    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls podName:9dfc4909-2e6d-4b80-a781-ed7e8fa59616 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:00.508817557 +0000 UTC m=+35.030986593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls") pod "dns-default-jf9x8" (UID: "9dfc4909-2e6d-4b80-a781-ed7e8fa59616") : secret "dns-default-metrics-tls" not found
Sep 10 12:30:59.994698 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.994664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:30:59.995344 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.994664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:30:59.995344 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.994664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:30:59.998493 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.998471    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 12:30:59.998626 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.998494    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sz48d\""
Sep 10 12:30:59.998626 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.998500    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ll527\""
Sep 10 12:30:59.998626 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.998525    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 12:30:59.998626 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.998574    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 12:30:59.998850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:30:59.998639    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 12:31:00.297631 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:00.297579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xthlv" event={"ID":"ee744d16-fa03-4d53-98fe-cb6fea265c0c","Type":"ContainerStarted","Data":"833d5be9264e65a4c4c83f5fc541a7902d60cc98a93e7a7927bdbd5d09254600"}
Sep 10 12:31:00.501155 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:00.501119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:00.516570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:00.516542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:31:00.516751 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:00.516725    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:00.516817 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:00.516806    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls podName:9dfc4909-2e6d-4b80-a781-ed7e8fa59616 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:02.516783295 +0000 UTC m=+37.038952325 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls") pod "dns-default-jf9x8" (UID: "9dfc4909-2e6d-4b80-a781-ed7e8fa59616") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:00.602883 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:00.602812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:02.531201 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:02.531161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:31:02.531854 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:02.531367    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:02.531854 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:02.531453    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls podName:9dfc4909-2e6d-4b80-a781-ed7e8fa59616 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:06.531431084 +0000 UTC m=+41.053600137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls") pod "dns-default-jf9x8" (UID: "9dfc4909-2e6d-4b80-a781-ed7e8fa59616") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:03.308955 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:03.308920    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3193d5-30ab-4869-8ec9-ad6af393c727" containerID="6f9c3683d011eba14a64c5fef920e3783be894e7690fafdca58af475c83ed484" exitCode=0
Sep 10 12:31:03.309105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:03.308970    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" event={"ID":"af3193d5-30ab-4869-8ec9-ad6af393c727","Type":"ContainerDied","Data":"6f9c3683d011eba14a64c5fef920e3783be894e7690fafdca58af475c83ed484"}
Sep 10 12:31:03.310582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:03.310559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xthlv" event={"ID":"ee744d16-fa03-4d53-98fe-cb6fea265c0c","Type":"ContainerStarted","Data":"0cbb6d053200e8ab6c0d4831fdf1bdec0eef7fbd92ef4b1fac4cde5672a489c5"}
Sep 10 12:31:03.348409 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:03.348354    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xthlv" podStartSLOduration=1.778128404 podStartE2EDuration="5.34833337s" totalImagesPullingTime="3.570204966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:03.347158913 +0000 UTC m=+37.869327966" watchObservedRunningTime="2026-09-10 12:31:03.34833337 +0000 UTC m=+37.870502423"
Sep 10 12:31:04.064712 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:04.064668    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:04.316573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:04.316490    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3193d5-30ab-4869-8ec9-ad6af393c727" containerID="c3b19fdc50ff7b5091c66ca8f6d91f2b75e56862ebbd81a8ed84efe6cdb7ff11" exitCode=0
Sep 10 12:31:04.316828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:04.316568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" event={"ID":"af3193d5-30ab-4869-8ec9-ad6af393c727","Type":"ContainerDied","Data":"c3b19fdc50ff7b5091c66ca8f6d91f2b75e56862ebbd81a8ed84efe6cdb7ff11"}
Sep 10 12:31:05.321902 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:05.321871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" event={"ID":"af3193d5-30ab-4869-8ec9-ad6af393c727","Type":"ContainerStarted","Data":"83df67f49a9c4f1242ebdc3cc72a25578abd38d7c9786f1e33defce0f2e80d8b"}
Sep 10 12:31:05.346083 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:05.345947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fwl5z" podStartSLOduration=7.275576957 podStartE2EDuration="39.345929372s" totalImagesPullingTime="32.070352415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:05.344254269 +0000 UTC m=+39.866423330" watchObservedRunningTime="2026-09-10 12:31:05.345929372 +0000 UTC m=+39.868098425"
Sep 10 12:31:05.861411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:05.861371    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:06.557623 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:06.557574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:31:06.557999 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:06.557744    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:06.557999 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:06.557811    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls podName:9dfc4909-2e6d-4b80-a781-ed7e8fa59616 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:14.557793407 +0000 UTC m=+49.079962439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls") pod "dns-default-jf9x8" (UID: "9dfc4909-2e6d-4b80-a781-ed7e8fa59616") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:10.941576 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:10.941494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:11.023101 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:11.023076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:14.606901 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:14.606855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:31:14.607287 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:14.607012    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:14.607287 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:14.607079    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls podName:9dfc4909-2e6d-4b80-a781-ed7e8fa59616 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:30.607063694 +0000 UTC m=+65.129232725 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls") pod "dns-default-jf9x8" (UID: "9dfc4909-2e6d-4b80-a781-ed7e8fa59616") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:16.857168 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:16.857130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:18.821660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:18.821628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:18.891428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:18.891389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:25.289809 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:25.289781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftz99"
Sep 10 12:31:25.480084 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:25.480053    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:25.549798 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:25.549734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:26.065230 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:26.065198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:26.158840 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:26.158812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:26.228903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:26.228875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:30.616696 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.616656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:31:30.616696 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.616694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:31:30.617172 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:30.616804    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:30.617172 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:30.616861    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls podName:9dfc4909-2e6d-4b80-a781-ed7e8fa59616 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:02.616845762 +0000 UTC m=+97.139014816 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls") pod "dns-default-jf9x8" (UID: "9dfc4909-2e6d-4b80-a781-ed7e8fa59616") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:30.620724 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.620703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 12:31:30.630258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.630241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/646110b4-56d3-44b3-b75c-3fb2883b274b-original-pull-secret\") pod \"global-pull-secret-syncer-n4575\" (UID: \"646110b4-56d3-44b3-b75c-3fb2883b274b\") " pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:31:30.717459 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.717434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:31:30.720305 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.720288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 12:31:30.728059 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:30.728043    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 12:31:30.728106 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:30.728093    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs podName:2eb054c1-3406-4776-b4b6-b44b6d1f9849 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:34.728078745 +0000 UTC m=+129.250247774 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs") pod "network-metrics-daemon-dj27m" (UID: "2eb054c1-3406-4776-b4b6-b44b6d1f9849") : secret "metrics-daemon-secret" not found
Sep 10 12:31:30.817876 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.817853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:31:30.820674 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.820661    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 12:31:30.831792 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.831775    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 12:31:30.841359 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.841334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjvx\" (UniqueName: \"kubernetes.io/projected/63fb7115-8a68-4592-b052-c9b1507c45ff-kube-api-access-hcjvx\") pod \"network-check-target-btxk6\" (UID: \"63fb7115-8a68-4592-b052-c9b1507c45ff\") " pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:31:30.913300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.913254    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ll527\""
Sep 10 12:31:30.915339 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.915325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n4575"
Sep 10 12:31:30.921321 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:30.921308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:31:31.194890 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:31.194819    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n4575"]
Sep 10 12:31:31.198279 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:31:31.198259    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646110b4_56d3_44b3_b75c_3fb2883b274b.slice/crio-36592d5f1e61ab3909cefd3c64a3a7a733e93382a31db4d0dd8cf067320a36db WatchSource:0}: Error finding container 36592d5f1e61ab3909cefd3c64a3a7a733e93382a31db4d0dd8cf067320a36db: Status 404 returned error can't find the container with id 36592d5f1e61ab3909cefd3c64a3a7a733e93382a31db4d0dd8cf067320a36db
Sep 10 12:31:31.200222 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:31.200196    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-btxk6"]
Sep 10 12:31:31.202906 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:31:31.202881    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fb7115_8a68_4592_b052_c9b1507c45ff.slice/crio-4b85ff020a7bb845edf6c27fa181d5feb6bea60856a6dc377ee4f445345f76f6 WatchSource:0}: Error finding container 4b85ff020a7bb845edf6c27fa181d5feb6bea60856a6dc377ee4f445345f76f6: Status 404 returned error can't find the container with id 4b85ff020a7bb845edf6c27fa181d5feb6bea60856a6dc377ee4f445345f76f6
Sep 10 12:31:31.383074 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:31.383044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n4575" event={"ID":"646110b4-56d3-44b3-b75c-3fb2883b274b","Type":"ContainerStarted","Data":"36592d5f1e61ab3909cefd3c64a3a7a733e93382a31db4d0dd8cf067320a36db"}
Sep 10 12:31:31.384124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:31.384096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-btxk6" event={"ID":"63fb7115-8a68-4592-b052-c9b1507c45ff","Type":"ContainerStarted","Data":"4b85ff020a7bb845edf6c27fa181d5feb6bea60856a6dc377ee4f445345f76f6"}
Sep 10 12:31:36.400423 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:36.400333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-btxk6" event={"ID":"63fb7115-8a68-4592-b052-c9b1507c45ff","Type":"ContainerStarted","Data":"f33d661ee8ae84c858906e83fadd9d14f05c91dadee0093dc917886569f91739"}
Sep 10 12:31:36.400891 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:36.400435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:31:36.401842 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:36.401807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n4575" event={"ID":"646110b4-56d3-44b3-b75c-3fb2883b274b","Type":"ContainerStarted","Data":"605c64b9d73d6baf579322fe2456bda8154d9461ef9a51c3de75003d28c1e9d3"}
Sep 10 12:31:36.419840 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:36.419797    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-btxk6" podStartSLOduration=65.501714782 podStartE2EDuration="1m10.419784889s" totalImagesPullingTime="4.918070107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:36.419228969 +0000 UTC m=+70.941398043" watchObservedRunningTime="2026-09-10 12:31:36.419784889 +0000 UTC m=+70.941953940"
Sep 10 12:31:36.440467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:36.440427    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n4575" podStartSLOduration=65.518731242 podStartE2EDuration="1m10.440414209s" totalImagesPullingTime="4.921682967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:36.440150226 +0000 UTC m=+70.962319279" watchObservedRunningTime="2026-09-10 12:31:36.440414209 +0000 UTC m=+70.962583285"
Sep 10 12:31:41.554940 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.554907    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx"]
Sep 10 12:31:41.557115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.557095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx"
Sep 10 12:31:41.558450 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.558423    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"]
Sep 10 12:31:41.559935 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.559912    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cpffp\""
Sep 10 12:31:41.560027 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.560016    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:31:41.560372 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.560351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:41.561307 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.561288    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 12:31:41.563957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.563938    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84884d55d8-8jzhd"]
Sep 10 12:31:41.564995 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.564972    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:31:41.567261 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.567234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tntmd\""
Sep 10 12:31:41.567755 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.567728    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 12:31:41.568056 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.568039    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 12:31:41.569540 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.569518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.571726 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.571540    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-dockercfg-wtkhq\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wtkhq\"" type="*v1.Secret"
Sep 10 12:31:41.571826 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.571748    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-stats-default\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret"
Sep 10 12:31:41.571889 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.571858    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-ingress-cert\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret"
Sep 10 12:31:41.572050 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.572025    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 10 12:31:41.572153 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.572071    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-bundle\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap"
Sep 10 12:31:41.572403 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.572378    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 10 12:31:41.572480 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.572463    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx"]
Sep 10 12:31:41.572782 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.572762    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 12:31:41.577549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.577528    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"]
Sep 10 12:31:41.583642 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.583622    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84884d55d8-8jzhd"]
Sep 10 12:31:41.592144 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.592122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669fp\" (UniqueName: \"kubernetes.io/projected/d0475af5-553d-4372-a2e1-0353f8f96578-kube-api-access-669fp\") pod \"volume-data-source-validator-695c895bd6-57crx\" (UID: \"d0475af5-553d-4372-a2e1-0353f8f96578\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx"
Sep 10 12:31:41.592259 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.592150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-stats-auth\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.592259 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.592166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjcvt\" (UniqueName: \"kubernetes.io/projected/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-kube-api-access-qjcvt\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.592259 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.592184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.592259 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.592255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.592380 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.592287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-default-certificate\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.592380 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.592303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvvv\" (UniqueName: \"kubernetes.io/projected/d5bb1fc1-9166-48a0-9d24-07b83908e662-kube-api-access-lzvvv\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:41.592380 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.592327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:41.680271 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.680249    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-9kcpx"]
Sep 10 12:31:41.682057 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.682043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.685943 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.685926    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:31:41.686955 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.686915    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 12:31:41.687691 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.687517    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 12:31:41.687970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.687947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 12:31:41.688317 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.688300    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zvs8n\""
Sep 10 12:31:41.692627 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692609    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 12:31:41.692702 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.692746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-default-certificate\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.692746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvvv\" (UniqueName: \"kubernetes.io/projected/d5bb1fc1-9166-48a0-9d24-07b83908e662-kube-api-access-lzvvv\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:41.692831 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a71518e-b89a-40af-bd74-3c444b7cabe6-serving-cert\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.692831 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.692778    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 12:31:41.692831 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a71518e-b89a-40af-bd74-3c444b7cabe6-trusted-ca\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.692939 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.692841    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:42.192825944 +0000 UTC m=+76.714994974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : secret "router-metrics-certs-default" not found
Sep 10 12:31:41.692939 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:41.692939 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669fp\" (UniqueName: \"kubernetes.io/projected/d0475af5-553d-4372-a2e1-0353f8f96578-kube-api-access-669fp\") pod \"volume-data-source-validator-695c895bd6-57crx\" (UID: \"d0475af5-553d-4372-a2e1-0353f8f96578\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx"
Sep 10 12:31:41.693076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-stats-auth\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.693076 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.692959    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 12:31:41.693076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.692969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcvt\" (UniqueName: \"kubernetes.io/projected/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-kube-api-access-qjcvt\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.693076 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:41.693002    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls podName:d5bb1fc1-9166-48a0-9d24-07b83908e662 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:42.192987463 +0000 UTC m=+76.715156507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2wbwm" (UID: "d5bb1fc1-9166-48a0-9d24-07b83908e662") : secret "samples-operator-tls" not found
Sep 10 12:31:41.693076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.693022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a71518e-b89a-40af-bd74-3c444b7cabe6-config\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.693076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.693047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79msz\" (UniqueName: \"kubernetes.io/projected/1a71518e-b89a-40af-bd74-3c444b7cabe6-kube-api-access-79msz\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.693274 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.693083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:41.700186 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.700165    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-9kcpx"]
Sep 10 12:31:41.710605 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.710575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-669fp\" (UniqueName: \"kubernetes.io/projected/d0475af5-553d-4372-a2e1-0353f8f96578-kube-api-access-669fp\") pod \"volume-data-source-validator-695c895bd6-57crx\" (UID: \"d0475af5-553d-4372-a2e1-0353f8f96578\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx"
Sep 10 12:31:41.714742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.714723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvvv\" (UniqueName: \"kubernetes.io/projected/d5bb1fc1-9166-48a0-9d24-07b83908e662-kube-api-access-lzvvv\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:41.793550 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.793521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a71518e-b89a-40af-bd74-3c444b7cabe6-config\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.793550 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.793549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79msz\" (UniqueName: \"kubernetes.io/projected/1a71518e-b89a-40af-bd74-3c444b7cabe6-kube-api-access-79msz\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.793758 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.793690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a71518e-b89a-40af-bd74-3c444b7cabe6-serving-cert\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.793758 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.793733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a71518e-b89a-40af-bd74-3c444b7cabe6-trusted-ca\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.794153 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.794127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a71518e-b89a-40af-bd74-3c444b7cabe6-config\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.794491 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.794469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a71518e-b89a-40af-bd74-3c444b7cabe6-trusted-ca\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.795860 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.795842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a71518e-b89a-40af-bd74-3c444b7cabe6-serving-cert\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.805952 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.805901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79msz\" (UniqueName: \"kubernetes.io/projected/1a71518e-b89a-40af-bd74-3c444b7cabe6-kube-api-access-79msz\") pod \"console-operator-85b454874b-9kcpx\" (UID: \"1a71518e-b89a-40af-bd74-3c444b7cabe6\") " pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:41.871679 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.871656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx"
Sep 10 12:31:41.990699 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:41.990671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:42.098132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.098065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:42.139486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.139456    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx"]
Sep 10 12:31:42.143548 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:31:42.143523    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0475af5_553d_4372_a2e1_0353f8f96578.slice/crio-c148673702e14327b8804adde30e78275c7b4f676f0b9210cef04a93c4c88c4b WatchSource:0}: Error finding container c148673702e14327b8804adde30e78275c7b4f676f0b9210cef04a93c4c88c4b: Status 404 returned error can't find the container with id c148673702e14327b8804adde30e78275c7b4f676f0b9210cef04a93c4c88c4b
Sep 10 12:31:42.196477 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.196454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:42.196576 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.196499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:42.196664 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:42.196590    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 12:31:42.196664 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:42.196627    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 12:31:42.196768 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:42.196665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls podName:d5bb1fc1-9166-48a0-9d24-07b83908e662 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.196650946 +0000 UTC m=+77.718819975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2wbwm" (UID: "d5bb1fc1-9166-48a0-9d24-07b83908e662") : secret "samples-operator-tls" not found
Sep 10 12:31:42.196768 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:42.196678    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.196672565 +0000 UTC m=+77.718841595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : secret "router-metrics-certs-default" not found
Sep 10 12:31:42.199945 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.199922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:42.259149 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.259128    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-9kcpx"]
Sep 10 12:31:42.262006 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:31:42.261985    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a71518e_b89a_40af_bd74_3c444b7cabe6.slice/crio-59b0f336f079b95db70f649dfdb73e8798b96de9aaf3c93cb0dee45ec746e5cd WatchSource:0}: Error finding container 59b0f336f079b95db70f649dfdb73e8798b96de9aaf3c93cb0dee45ec746e5cd: Status 404 returned error can't find the container with id 59b0f336f079b95db70f649dfdb73e8798b96de9aaf3c93cb0dee45ec746e5cd
Sep 10 12:31:42.376889 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.376825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 12:31:42.385355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.385334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-stats-auth\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:42.417934 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.417906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-9kcpx" event={"ID":"1a71518e-b89a-40af-bd74-3c444b7cabe6","Type":"ContainerStarted","Data":"59b0f336f079b95db70f649dfdb73e8798b96de9aaf3c93cb0dee45ec746e5cd"}
Sep 10 12:31:42.419063 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.419042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx" event={"ID":"d0475af5-553d-4372-a2e1-0353f8f96578","Type":"ContainerStarted","Data":"c148673702e14327b8804adde30e78275c7b4f676f0b9210cef04a93c4c88c4b"}
Sep 10 12:31:42.498417 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.498393    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 12:31:42.505838 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:42.505814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-default-certificate\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:42.693744 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:42.693665    2577 configmap.go:193] Couldn't get configMap openshift-ingress/service-ca-bundle: failed to sync configmap cache: timed out waiting for the condition
Sep 10 12:31:42.694120 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:42.693762    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.193743726 +0000 UTC m=+77.715912775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 12:31:42.712436 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:42.712402    2577 projected.go:291] Couldn't get configMap openshift-ingress/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 10 12:31:43.052940 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.052913    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 12:31:43.062956 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:43.062924    2577 projected.go:196] Error preparing data for projected volume kube-api-access-qjcvt for pod openshift-ingress/router-default-84884d55d8-8jzhd: failed to sync configmap cache: timed out waiting for the condition
Sep 10 12:31:43.063102 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:43.063016    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-kube-api-access-qjcvt podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.562993461 +0000 UTC m=+78.085162507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qjcvt" (UniqueName: "kubernetes.io/projected/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-kube-api-access-qjcvt") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 12:31:43.066312 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.066270    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 12:31:43.116777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.116751    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 12:31:43.139272 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.139248    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wtkhq\""
Sep 10 12:31:43.205034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.205006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:43.205154 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.205114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:43.205265 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:43.205248    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 12:31:43.205326 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:43.205254    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:44.205240826 +0000 UTC m=+78.727409855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : configmap references non-existent config key: service-ca.crt
Sep 10 12:31:43.205326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.205305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:43.205418 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:43.205355    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls podName:d5bb1fc1-9166-48a0-9d24-07b83908e662 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:45.20533825 +0000 UTC m=+79.727507279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2wbwm" (UID: "d5bb1fc1-9166-48a0-9d24-07b83908e662") : secret "samples-operator-tls" not found
Sep 10 12:31:43.205458 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:43.205411    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 12:31:43.205502 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:43.205467    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:45.205452597 +0000 UTC m=+79.727621642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : secret "router-metrics-certs-default" not found
Sep 10 12:31:43.424228 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.424137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx" event={"ID":"d0475af5-553d-4372-a2e1-0353f8f96578","Type":"ContainerStarted","Data":"1f56bb4cc6b86087eabd6b864ea1836f153b5ca35faec8d45bee85a4c6934ea7"}
Sep 10 12:31:43.445193 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.445141    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-57crx" podStartSLOduration=1.36369602 podStartE2EDuration="2.44512702s" totalImagesPullingTime="1.081431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:43.444152997 +0000 UTC m=+77.966322049" watchObservedRunningTime="2026-09-10 12:31:43.44512702 +0000 UTC m=+77.967296101"
Sep 10 12:31:43.554215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.554187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:43.608549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.608520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcvt\" (UniqueName: \"kubernetes.io/projected/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-kube-api-access-qjcvt\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:43.611021 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.610994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjcvt\" (UniqueName: \"kubernetes.io/projected/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-kube-api-access-qjcvt\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:43.654665 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:43.654636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:44.212818 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:44.212792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:44.213081 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:44.212929    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:46.21291084 +0000 UTC m=+80.735079873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : configmap references non-existent config key: service-ca.crt
Sep 10 12:31:44.429099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:44.429038    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/0.log"
Sep 10 12:31:44.429099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:44.429078    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a71518e-b89a-40af-bd74-3c444b7cabe6" containerID="ac707f689cb2dd23c2b979a38feca5687b0ee76f7059c7151c7282f2bad56add" exitCode=255
Sep 10 12:31:44.429335 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:44.429163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-9kcpx" event={"ID":"1a71518e-b89a-40af-bd74-3c444b7cabe6","Type":"ContainerDied","Data":"ac707f689cb2dd23c2b979a38feca5687b0ee76f7059c7151c7282f2bad56add"}
Sep 10 12:31:44.429434 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:44.429418    2577 scope.go:117] "RemoveContainer" containerID="ac707f689cb2dd23c2b979a38feca5687b0ee76f7059c7151c7282f2bad56add"
Sep 10 12:31:45.060178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.060150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:45.095735 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.095707    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9"]
Sep 10 12:31:45.097648 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.097631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9"
Sep 10 12:31:45.100062 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.100034    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fwkn9\""
Sep 10 12:31:45.101144 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.101129    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 12:31:45.101225 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.101183    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 12:31:45.106017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.105999    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9"]
Sep 10 12:31:45.120022 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.120004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwd4\" (UniqueName: \"kubernetes.io/projected/1e5c1239-32f3-4b59-a5f5-db054c5c4575-kube-api-access-pfwd4\") pod \"migrator-fdbb8fdff-7vqc9\" (UID: \"1e5c1239-32f3-4b59-a5f5-db054c5c4575\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9"
Sep 10 12:31:45.150734 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.150698    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:45.221283 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.221246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwd4\" (UniqueName: \"kubernetes.io/projected/1e5c1239-32f3-4b59-a5f5-db054c5c4575-kube-api-access-pfwd4\") pod \"migrator-fdbb8fdff-7vqc9\" (UID: \"1e5c1239-32f3-4b59-a5f5-db054c5c4575\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9"
Sep 10 12:31:45.221726 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.221305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:45.221726 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.221349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:45.221726 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.221457    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 12:31:45.221726 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.221511    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls podName:d5bb1fc1-9166-48a0-9d24-07b83908e662 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:49.221497459 +0000 UTC m=+83.743666489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2wbwm" (UID: "d5bb1fc1-9166-48a0-9d24-07b83908e662") : secret "samples-operator-tls" not found
Sep 10 12:31:45.221726 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.221458    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 12:31:45.221726 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.221590    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:49.221577158 +0000 UTC m=+83.743746201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : secret "router-metrics-certs-default" not found
Sep 10 12:31:45.231220 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.231198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwd4\" (UniqueName: \"kubernetes.io/projected/1e5c1239-32f3-4b59-a5f5-db054c5c4575-kube-api-access-pfwd4\") pod \"migrator-fdbb8fdff-7vqc9\" (UID: \"1e5c1239-32f3-4b59-a5f5-db054c5c4575\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9"
Sep 10 12:31:45.250129 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.250098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:45.406860 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.406778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9"
Sep 10 12:31:45.434186 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.434157    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:31:45.434640 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.434625    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/0.log"
Sep 10 12:31:45.434709 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.434660    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a71518e-b89a-40af-bd74-3c444b7cabe6" containerID="0c3d41da5c063c68f34adb6fed7749103766eeac6047f1d750a031ee3f59ae69" exitCode=255
Sep 10 12:31:45.434746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.434712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-9kcpx" event={"ID":"1a71518e-b89a-40af-bd74-3c444b7cabe6","Type":"ContainerDied","Data":"0c3d41da5c063c68f34adb6fed7749103766eeac6047f1d750a031ee3f59ae69"}
Sep 10 12:31:45.434791 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.434780    2577 scope.go:117] "RemoveContainer" containerID="ac707f689cb2dd23c2b979a38feca5687b0ee76f7059c7151c7282f2bad56add"
Sep 10 12:31:45.435004 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.434982    2577 scope.go:117] "RemoveContainer" containerID="0c3d41da5c063c68f34adb6fed7749103766eeac6047f1d750a031ee3f59ae69"
Sep 10 12:31:45.435195 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.435171    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-9kcpx_openshift-console-operator(1a71518e-b89a-40af-bd74-3c444b7cabe6)\"" pod="openshift-console-operator/console-operator-85b454874b-9kcpx" podUID="1a71518e-b89a-40af-bd74-3c444b7cabe6"
Sep 10 12:31:45.615487 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.615459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:45.652693 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.652653    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2nh2p"]
Sep 10 12:31:45.655125 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.655110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.657878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.657812    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 12:31:45.662051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.662030    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 12:31:45.662051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.662050    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 12:31:45.663012 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.662997    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qvkkn\""
Sep 10 12:31:45.663244 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.663229    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 12:31:45.672648 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.672629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2nh2p"]
Sep 10 12:31:45.697124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.697104    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9"]
Sep 10 12:31:45.699424 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:31:45.699403    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5c1239_32f3_4b59_a5f5_db054c5c4575.slice/crio-9a2627ca8464fcba997a588cbefbf8d0f064d40492572dd54c6c6a4229c757be WatchSource:0}: Error finding container 9a2627ca8464fcba997a588cbefbf8d0f064d40492572dd54c6c6a4229c757be: Status 404 returned error can't find the container with id 9a2627ca8464fcba997a588cbefbf8d0f064d40492572dd54c6c6a4229c757be
Sep 10 12:31:45.707242 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.707222    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:45.726512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.726489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-data-volume\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.726619 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.726521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.726619 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.726543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbw66\" (UniqueName: \"kubernetes.io/projected/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-api-access-lbw66\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.726619 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.726574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-crio-socket\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.726729 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.726696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.827389 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.827364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.827445 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.827395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-data-volume\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.827445 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.827411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.827445 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.827429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbw66\" (UniqueName: \"kubernetes.io/projected/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-api-access-lbw66\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.827584 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.827456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-crio-socket\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.827584 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.827496    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:45.827584 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.827544    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:31:46.327529996 +0000 UTC m=+80.849699026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:45.827584 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.827497    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:45.827584 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.827560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-crio-socket\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.827868 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:45.827637    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:31:46.327619776 +0000 UTC m=+80.849788824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:45.827868 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.827825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-data-volume\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:45.843533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:45.843507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbw66\" (UniqueName: \"kubernetes.io/projected/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-api-access-lbw66\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:46.230499 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:46.230470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:46.230835 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:46.230588    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:50.230574463 +0000 UTC m=+84.752743498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : configmap references non-existent config key: service-ca.crt
Sep 10 12:31:46.331511 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:46.331482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:46.331661 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:46.331518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:46.331661 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:46.331640    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:46.331757 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:46.331688    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:31:47.33167436 +0000 UTC m=+81.853843390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:46.331757 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:46.331640    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:46.331841 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:46.331779    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:31:47.331766618 +0000 UTC m=+81.853935653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:46.427142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:46.427115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:46.439942 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:46.439921    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:31:46.440344 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:46.440325    2577 scope.go:117] "RemoveContainer" containerID="0c3d41da5c063c68f34adb6fed7749103766eeac6047f1d750a031ee3f59ae69"
Sep 10 12:31:46.440524 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:46.440499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-9kcpx_openshift-console-operator(1a71518e-b89a-40af-bd74-3c444b7cabe6)\"" pod="openshift-console-operator/console-operator-85b454874b-9kcpx" podUID="1a71518e-b89a-40af-bd74-3c444b7cabe6"
Sep 10 12:31:46.441486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:46.441440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9" event={"ID":"1e5c1239-32f3-4b59-a5f5-db054c5c4575","Type":"ContainerStarted","Data":"9a2627ca8464fcba997a588cbefbf8d0f064d40492572dd54c6c6a4229c757be"}
Sep 10 12:31:46.519729 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:46.519694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:47.338165 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:47.338128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:47.338563 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:47.338174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:47.338563 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:47.338241    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:47.338563 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:47.338276    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:47.338563 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:47.338289    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:31:49.338276309 +0000 UTC m=+83.860445339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:47.338563 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:47.338425    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:31:49.338407966 +0000 UTC m=+83.860576997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:47.447023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:47.446989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9" event={"ID":"1e5c1239-32f3-4b59-a5f5-db054c5c4575","Type":"ContainerStarted","Data":"013e0483ce67046b822d0be470f52bc848926aa9f4d1f61819b0056266066da3"}
Sep 10 12:31:47.447150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:47.447040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9" event={"ID":"1e5c1239-32f3-4b59-a5f5-db054c5c4575","Type":"ContainerStarted","Data":"f8ab9acc36b5325880b28f8981b4b21807ce4d3f546fa12a6e74686bcb1ad9be"}
Sep 10 12:31:47.470106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:47.470063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7vqc9" podStartSLOduration=1.26305025 podStartE2EDuration="2.470048693s" totalImagesPullingTime="1.206998443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:47.469690254 +0000 UTC m=+81.991859326" watchObservedRunningTime="2026-09-10 12:31:47.470048693 +0000 UTC m=+81.992217744"
Sep 10 12:31:47.569877 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:47.569828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:47.672203 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:47.672128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:48.312488 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:48.312453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:48.337313 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:48.337292    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-59cz5_72c9ae18-ee7a-4949-b2ae-4de97e29e46a/dns-node-resolver/0.log"
Sep 10 12:31:48.373879 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:48.373854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:48.953186 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:48.953152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:49.253515 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:49.253430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:49.253688 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:49.253555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:49.253688 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:49.253586    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 12:31:49.253688 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:49.253660    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 12:31:49.253688 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:49.253674    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls podName:d5bb1fc1-9166-48a0-9d24-07b83908e662 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:57.253654587 +0000 UTC m=+91.775823630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2wbwm" (UID: "d5bb1fc1-9166-48a0-9d24-07b83908e662") : secret "samples-operator-tls" not found
Sep 10 12:31:49.253833 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:49.253695    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:57.253685041 +0000 UTC m=+91.775854080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : secret "router-metrics-certs-default" not found
Sep 10 12:31:49.354768 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:49.354730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:49.354919 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:49.354841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:49.354919 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:49.354857    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:49.354988 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:49.354920    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:31:53.354905445 +0000 UTC m=+87.877074490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:49.354988 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:49.354935    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:49.354988 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:49.354977    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:31:53.354962855 +0000 UTC m=+87.877131885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:49.538278 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:49.538250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rrqld_2a86ebb5-e828-4641-9ffd-c25ab4039f7e/node-ca/0.log"
Sep 10 12:31:50.260245 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:50.260205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:50.260424 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:50.260405    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle podName:029b95e7-f063-4791-9a5b-1f3db7ad0c3a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:58.260386202 +0000 UTC m=+92.782555235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle") pod "router-default-84884d55d8-8jzhd" (UID: "029b95e7-f063-4791-9a5b-1f3db7ad0c3a") : configmap references non-existent config key: service-ca.crt
Sep 10 12:31:50.339605 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:50.339567    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xthlv_ee744d16-fa03-4d53-98fe-cb6fea265c0c/serve-healthcheck-canary/0.log"
Sep 10 12:31:50.479571 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:50.479540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:50.539087 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:50.539067    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-7vqc9_1e5c1239-32f3-4b59-a5f5-db054c5c4575/migrator/0.log"
Sep 10 12:31:50.611875 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:50.611845    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:50.725825 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:50.725795    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:50.754860 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:50.754839    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-7vqc9_1e5c1239-32f3-4b59-a5f5-db054c5c4575/graceful-termination/0.log"
Sep 10 12:31:50.937325 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:50.937251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:51.167047 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:51.167018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:51.991088 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:51.991061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:51.991455 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:51.991109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:31:51.991455 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:51.991413    2577 scope.go:117] "RemoveContainer" containerID="0c3d41da5c063c68f34adb6fed7749103766eeac6047f1d750a031ee3f59ae69"
Sep 10 12:31:51.991563 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:51.991545    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-9kcpx_openshift-console-operator(1a71518e-b89a-40af-bd74-3c444b7cabe6)\"" pod="openshift-console-operator/console-operator-85b454874b-9kcpx" podUID="1a71518e-b89a-40af-bd74-3c444b7cabe6"
Sep 10 12:31:52.363708 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:52.363677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:52.572517 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:52.572486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:52.647088 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:52.647026    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:52.767206 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:52.767178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:52.964028 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:52.963952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:53.166387 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:53.166357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:53.383318 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:53.383287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:53.383318 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:53.383322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:31:53.383515 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:53.383420    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:53.383515 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:53.383430    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:53.383515 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:53.383480    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:32:01.383466601 +0000 UTC m=+95.905635635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:53.383515 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:31:53.383493    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls podName:52e6ec7a-6c26-4bea-a9da-7d29d4e036fc nodeName:}" failed. No retries permitted until 2026-09-10 12:32:01.383487235 +0000 UTC m=+95.905656265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nh2p" (UID: "52e6ec7a-6c26-4bea-a9da-7d29d4e036fc") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:54.091143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:54.091108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:54.193701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:54.193670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:54.257341 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:54.257316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:54.564765 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:54.564734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:54.765368 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:54.765335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:54.967553 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:54.967476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:55.038480 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:55.038453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:55.362428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:55.362400    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:55.762575 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:55.762494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:56.376044 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:56.376008    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:56.760212 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:56.760128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:56.860165 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:56.860136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:56.963910 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:56.963883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:57.310692 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:57.310655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:57.310852 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:57.310713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:57.313290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:57.313266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-metrics-certs\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:57.313290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:57.313284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5bb1fc1-9166-48a0-9d24-07b83908e662-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2wbwm\" (UID: \"d5bb1fc1-9166-48a0-9d24-07b83908e662\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:57.370342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:57.370319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:57.477558 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:57.477535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"
Sep 10 12:31:57.751161 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:57.751075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm"]
Sep 10 12:31:57.767684 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:57.767655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:58.264733 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.264697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:58.319869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.319840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:58.320448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.320425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029b95e7-f063-4791-9a5b-1f3db7ad0c3a-service-ca-bundle\") pod \"router-default-84884d55d8-8jzhd\" (UID: \"029b95e7-f063-4791-9a5b-1f3db7ad0c3a\") " pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:58.365400 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.365363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:58.383366 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.383340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:31:58.480065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.480035    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm" event={"ID":"d5bb1fc1-9166-48a0-9d24-07b83908e662","Type":"ContainerStarted","Data":"eb2f2cfaa12a9624a319e6dd7113450be90b84d47d83574e19e40d484b1466f9"}
Sep 10 12:31:58.677801 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.677768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84884d55d8-8jzhd"]
Sep 10 12:31:58.681229 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:31:58.681205    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029b95e7_f063_4791_9a5b_1f3db7ad0c3a.slice/crio-7810adfd634c9752ea2b9c4bde6d973d04633c310f712df990a50a7b1f2c091f WatchSource:0}: Error finding container 7810adfd634c9752ea2b9c4bde6d973d04633c310f712df990a50a7b1f2c091f: Status 404 returned error can't find the container with id 7810adfd634c9752ea2b9c4bde6d973d04633c310f712df990a50a7b1f2c091f
Sep 10 12:31:58.768754 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.768721    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:58.967441 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:58.967367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:59.484690 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:59.484653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84884d55d8-8jzhd" event={"ID":"029b95e7-f063-4791-9a5b-1f3db7ad0c3a","Type":"ContainerStarted","Data":"d4e2203f72bb3f206a1975a3ece2ff2d2c003b16ad9b9aeb94d63883375f7cc1"}
Sep 10 12:31:59.485052 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:59.484700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84884d55d8-8jzhd" event={"ID":"029b95e7-f063-4791-9a5b-1f3db7ad0c3a","Type":"ContainerStarted","Data":"7810adfd634c9752ea2b9c4bde6d973d04633c310f712df990a50a7b1f2c091f"}
Sep 10 12:31:59.507591 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:59.507547    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-84884d55d8-8jzhd" podStartSLOduration=18.507529642 podStartE2EDuration="18.507529642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:59.506807064 +0000 UTC m=+94.028976115" watchObservedRunningTime="2026-09-10 12:31:59.507529642 +0000 UTC m=+94.029698694"
Sep 10 12:31:59.576362 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:59.576336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:31:59.985310 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:31:59.985279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:00.375073 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.375046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:00.384128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.384104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:32:00.386504 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.386487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:32:00.489018 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.488995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm" event={"ID":"d5bb1fc1-9166-48a0-9d24-07b83908e662","Type":"ContainerStarted","Data":"7f64eadca3462626efb4f7d56999881f165ef21c3ae5741635e0317877f1f7ff"}
Sep 10 12:32:00.489445 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.489033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm" event={"ID":"d5bb1fc1-9166-48a0-9d24-07b83908e662","Type":"ContainerStarted","Data":"f7b48aef122fdabc4f2ce16c3b33f45b01fb6b2ea71ebef24377de647f3d935a"}
Sep 10 12:32:00.489445 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.489205    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:32:00.490296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.490279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84884d55d8-8jzhd"
Sep 10 12:32:00.511103 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.511063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2wbwm" podStartSLOduration=17.764307003 podStartE2EDuration="19.511050984s" totalImagesPullingTime="1.746743981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:00.51039034 +0000 UTC m=+95.032559395" watchObservedRunningTime="2026-09-10 12:32:00.511050984 +0000 UTC m=+95.033220072"
Sep 10 12:32:00.767058 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.766983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:00.968931 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:00.968899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:01.187630 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:01.187582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:01.442401 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:01.442310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:32:01.442401 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:01.442351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:32:01.443040 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:01.443018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:32:01.444693 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:01.444675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52e6ec7a-6c26-4bea-a9da-7d29d4e036fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nh2p\" (UID: \"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc\") " pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:32:01.564841 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:01.564802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nh2p"
Sep 10 12:32:01.843124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:01.843081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2nh2p"]
Sep 10 12:32:01.846778 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:01.846746    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e6ec7a_6c26_4bea_a9da_7d29d4e036fc.slice/crio-fbe0f7bab96b8f9aef545a543912669f8287d7f17cc8cd4c5664c6643056683c WatchSource:0}: Error finding container fbe0f7bab96b8f9aef545a543912669f8287d7f17cc8cd4c5664c6643056683c: Status 404 returned error can't find the container with id fbe0f7bab96b8f9aef545a543912669f8287d7f17cc8cd4c5664c6643056683c
Sep 10 12:32:02.207397 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:02.207323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:02.501186 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:02.501093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nh2p" event={"ID":"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc","Type":"ContainerStarted","Data":"c1895b896a2263ca56676651dffb4c5b95fa76029dfa87d7e67a76eb8ab87c99"}
Sep 10 12:32:02.501359 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:02.501194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nh2p" event={"ID":"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc","Type":"ContainerStarted","Data":"fbe0f7bab96b8f9aef545a543912669f8287d7f17cc8cd4c5664c6643056683c"}
Sep 10 12:32:02.652235 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:02.652201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:32:02.654696 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:02.654671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfc4909-2e6d-4b80-a781-ed7e8fa59616-metrics-tls\") pod \"dns-default-jf9x8\" (UID: \"9dfc4909-2e6d-4b80-a781-ed7e8fa59616\") " pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:32:02.702752 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:02.702732    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgmkk\""
Sep 10 12:32:02.710636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:02.710616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:32:02.989434 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:02.989405    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jf9x8"]
Sep 10 12:32:02.992878 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:02.992845    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfc4909_2e6d_4b80_a781_ed7e8fa59616.slice/crio-1c0d27b7410549426540136174025b22419b4e6de81af3c58399a161d683df72 WatchSource:0}: Error finding container 1c0d27b7410549426540136174025b22419b4e6de81af3c58399a161d683df72: Status 404 returned error can't find the container with id 1c0d27b7410549426540136174025b22419b4e6de81af3c58399a161d683df72
Sep 10 12:32:03.506319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:03.506283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jf9x8" event={"ID":"9dfc4909-2e6d-4b80-a781-ed7e8fa59616","Type":"ContainerStarted","Data":"1c0d27b7410549426540136174025b22419b4e6de81af3c58399a161d683df72"}
Sep 10 12:32:03.508371 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:03.508341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nh2p" event={"ID":"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc","Type":"ContainerStarted","Data":"b7275b96e3cde95443ee042de78d3abf83d87295651e8821fb096e40d4ebfb82"}
Sep 10 12:32:04.514789 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:04.514751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nh2p" event={"ID":"52e6ec7a-6c26-4bea-a9da-7d29d4e036fc","Type":"ContainerStarted","Data":"044a9433aef4ca5865fd7e51dbbfc99d0a8a5e7d83d29b617b5062b75dbfcb7b"}
Sep 10 12:32:04.540227 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:04.540171    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2nh2p" podStartSLOduration=17.184288722 podStartE2EDuration="19.540149786s" totalImagesPullingTime="2.355861064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:31:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:04.540040895 +0000 UTC m=+99.062209948" watchObservedRunningTime="2026-09-10 12:32:04.540149786 +0000 UTC m=+99.062318840"
Sep 10 12:32:05.520197 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:05.520157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jf9x8" event={"ID":"9dfc4909-2e6d-4b80-a781-ed7e8fa59616","Type":"ContainerStarted","Data":"df0a64ff06c15d1fd64f940c78925f533045588b80f89fdd72813946d9e84ed2"}
Sep 10 12:32:05.520611 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:05.520226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jf9x8" event={"ID":"9dfc4909-2e6d-4b80-a781-ed7e8fa59616","Type":"ContainerStarted","Data":"bcaae79fb4c26f853c8975209210c49beadf362d470271218c3fd835ae40d8a4"}
Sep 10 12:32:05.520611 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:05.520328    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:32:06.994551 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:06.994516    2577 scope.go:117] "RemoveContainer" containerID="0c3d41da5c063c68f34adb6fed7749103766eeac6047f1d750a031ee3f59ae69"
Sep 10 12:32:07.406630 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.406589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-btxk6"
Sep 10 12:32:07.483553 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.483505    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jf9x8" podStartSLOduration=67.615745361 podStartE2EDuration="1m9.483490304s" totalImagesPullingTime="1.867744943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:05.543252527 +0000 UTC m=+100.065421578" watchObservedRunningTime="2026-09-10 12:32:07.483490304 +0000 UTC m=+102.005659391"
Sep 10 12:32:07.527792 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.527767    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:32:07.527917 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.527828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-9kcpx" event={"ID":"1a71518e-b89a-40af-bd74-3c444b7cabe6","Type":"ContainerStarted","Data":"a37282d755e2d28dbdd6834cd87fce2b10b54ac583d4241d56a598a7e9cab2f1"}
Sep 10 12:32:07.528104 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.528084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:32:07.532855 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.532833    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-9kcpx"
Sep 10 12:32:07.610496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.610438    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-9kcpx" podStartSLOduration=24.781895353 podStartE2EDuration="26.610418301s" totalImagesPullingTime="1.828522948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:07.60967994 +0000 UTC m=+102.131849009" watchObservedRunningTime="2026-09-10 12:32:07.610418301 +0000 UTC m=+102.132587370"
Sep 10 12:32:07.764935 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.764862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:07.855315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.855270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:07.962773 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:07.962743    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:08.086552 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:08.086521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:08.190706 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:08.190674    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:08.300956 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:08.300924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:08.376388 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:08.376330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:08.705874 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:08.705795    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:09.318309 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:09.318270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:09.539750 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:09.539715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:09.967456 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:09.967419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:10.410409 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.410378    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-tmlbj"]
Sep 10 12:32:10.432221 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.432195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:10.441780 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.441756    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-tmlbj"]
Sep 10 12:32:10.441915 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.441854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-tmlbj"
Sep 10 12:32:10.444532 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.444511    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 12:32:10.444702 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.444555    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 12:32:10.444984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.444960    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zczks\""
Sep 10 12:32:10.502200 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.502144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwtf\" (UniqueName: \"kubernetes.io/projected/fad3daa4-04a8-4bcb-a8a2-875b5262e073-kube-api-access-cmwtf\") pod \"downloads-85d98c998f-tmlbj\" (UID: \"fad3daa4-04a8-4bcb-a8a2-875b5262e073\") " pod="openshift-console/downloads-85d98c998f-tmlbj"
Sep 10 12:32:10.602878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.602850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwtf\" (UniqueName: \"kubernetes.io/projected/fad3daa4-04a8-4bcb-a8a2-875b5262e073-kube-api-access-cmwtf\") pod \"downloads-85d98c998f-tmlbj\" (UID: \"fad3daa4-04a8-4bcb-a8a2-875b5262e073\") " pod="openshift-console/downloads-85d98c998f-tmlbj"
Sep 10 12:32:10.607590 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.607565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:10.616447 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.616427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwtf\" (UniqueName: \"kubernetes.io/projected/fad3daa4-04a8-4bcb-a8a2-875b5262e073-kube-api-access-cmwtf\") pod \"downloads-85d98c998f-tmlbj\" (UID: \"fad3daa4-04a8-4bcb-a8a2-875b5262e073\") " pod="openshift-console/downloads-85d98c998f-tmlbj"
Sep 10 12:32:10.751046 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.751018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-tmlbj"
Sep 10 12:32:10.753871 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:10.753818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:11.071927 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:11.071903    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad3daa4_04a8_4bcb_a8a2_875b5262e073.slice/crio-e05d0499e94fb73c505a33ca9194af6898a54c59508aeab8c8d9b461e29e68eb WatchSource:0}: Error finding container e05d0499e94fb73c505a33ca9194af6898a54c59508aeab8c8d9b461e29e68eb: Status 404 returned error can't find the container with id e05d0499e94fb73c505a33ca9194af6898a54c59508aeab8c8d9b461e29e68eb
Sep 10 12:32:11.083647 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:11.083616    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-tmlbj"]
Sep 10 12:32:11.542100 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:11.542071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-tmlbj" event={"ID":"fad3daa4-04a8-4bcb-a8a2-875b5262e073","Type":"ContainerStarted","Data":"e05d0499e94fb73c505a33ca9194af6898a54c59508aeab8c8d9b461e29e68eb"}
Sep 10 12:32:13.402656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:13.402613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:13.501142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:13.501110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:15.525425 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:15.525398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jf9x8"
Sep 10 12:32:15.810071 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:15.809996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:15.905133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:15.905104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:16.006368 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.006331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:16.076643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.076557    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"]
Sep 10 12:32:16.078956 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.078930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"
Sep 10 12:32:16.085610 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.085570    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 12:32:16.087753 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.087725    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gztz2\""
Sep 10 12:32:16.103723 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.103700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"]
Sep 10 12:32:16.146604 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.146560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/64fb7a24-54f4-4086-b4fd-f502603a60c7-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-svb9h\" (UID: \"64fb7a24-54f4-4086-b4fd-f502603a60c7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"
Sep 10 12:32:16.247017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.246984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/64fb7a24-54f4-4086-b4fd-f502603a60c7-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-svb9h\" (UID: \"64fb7a24-54f4-4086-b4fd-f502603a60c7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"
Sep 10 12:32:16.247178 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:32:16.247116    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 12:32:16.247240 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:32:16.247186    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64fb7a24-54f4-4086-b4fd-f502603a60c7-tls-certificates podName:64fb7a24-54f4-4086-b4fd-f502603a60c7 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:16.747169536 +0000 UTC m=+111.269338579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/64fb7a24-54f4-4086-b4fd-f502603a60c7-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-svb9h" (UID: "64fb7a24-54f4-4086-b4fd-f502603a60c7") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 12:32:16.479502 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.479409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:16.750463 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.750377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/64fb7a24-54f4-4086-b4fd-f502603a60c7-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-svb9h\" (UID: \"64fb7a24-54f4-4086-b4fd-f502603a60c7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"
Sep 10 12:32:16.753191 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.753167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/64fb7a24-54f4-4086-b4fd-f502603a60c7-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-svb9h\" (UID: \"64fb7a24-54f4-4086-b4fd-f502603a60c7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"
Sep 10 12:32:16.991691 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:16.991659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"
Sep 10 12:32:17.291075 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:17.291040    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fb7a24_54f4_4086_b4fd_f502603a60c7.slice/crio-96365911e4073e73849bc52b56ec7854e0bd0445c98012e28c52177f3cb7c3cc WatchSource:0}: Error finding container 96365911e4073e73849bc52b56ec7854e0bd0445c98012e28c52177f3cb7c3cc: Status 404 returned error can't find the container with id 96365911e4073e73849bc52b56ec7854e0bd0445c98012e28c52177f3cb7c3cc
Sep 10 12:32:17.291319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.291300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"]
Sep 10 12:32:17.413797 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.413773    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c6f59cdd-2w6d8"]
Sep 10 12:32:17.416514 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.416488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.430174 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.430151    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 12:32:17.431224 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.431202    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 12:32:17.431398 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.431377    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 12:32:17.431398 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.431391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c6f59cdd-2w6d8"]
Sep 10 12:32:17.431570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.431445    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 12:32:17.431738 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.431722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7wrxb\""
Sep 10 12:32:17.433331 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.433313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 12:32:17.556891 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.556828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4g8\" (UniqueName: \"kubernetes.io/projected/f6169f56-1dc5-40a8-b629-7b6356066f8f-kube-api-access-sj4g8\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.556891 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.556864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-serving-cert\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.556891 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.556886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-config\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.557067 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.556901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-service-ca\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.557067 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.556938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-oauth-config\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.557067 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.557018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-oauth-serving-cert\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.563447 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.563423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h" event={"ID":"64fb7a24-54f4-4086-b4fd-f502603a60c7","Type":"ContainerStarted","Data":"96365911e4073e73849bc52b56ec7854e0bd0445c98012e28c52177f3cb7c3cc"}
Sep 10 12:32:17.657951 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.657925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-oauth-config\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.658072 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.657970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-oauth-serving-cert\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.658152 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.658131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4g8\" (UniqueName: \"kubernetes.io/projected/f6169f56-1dc5-40a8-b629-7b6356066f8f-kube-api-access-sj4g8\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.658210 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.658178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-serving-cert\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.658263 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.658217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-config\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.658263 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.658243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-service-ca\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.658802 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.658766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-oauth-serving-cert\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.658925 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.658909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-service-ca\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.659074 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.659049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-config\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.660911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.660878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-oauth-config\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.660992 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.660910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-serving-cert\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.681346 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.681318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4g8\" (UniqueName: \"kubernetes.io/projected/f6169f56-1dc5-40a8-b629-7b6356066f8f-kube-api-access-sj4g8\") pod \"console-85c6f59cdd-2w6d8\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") " pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:17.727229 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:17.727207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:18.004650 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:18.004461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c6f59cdd-2w6d8"]
Sep 10 12:32:18.008030 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:18.007975    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6169f56_1dc5_40a8_b629_7b6356066f8f.slice/crio-69f8aa822aeb9c04e3f6563dfdcc09fef09c24d923cdd47c3e4356c7062e9909 WatchSource:0}: Error finding container 69f8aa822aeb9c04e3f6563dfdcc09fef09c24d923cdd47c3e4356c7062e9909: Status 404 returned error can't find the container with id 69f8aa822aeb9c04e3f6563dfdcc09fef09c24d923cdd47c3e4356c7062e9909
Sep 10 12:32:18.303338 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:18.303303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:18.568823 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:18.568747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6f59cdd-2w6d8" event={"ID":"f6169f56-1dc5-40a8-b629-7b6356066f8f","Type":"ContainerStarted","Data":"69f8aa822aeb9c04e3f6563dfdcc09fef09c24d923cdd47c3e4356c7062e9909"}
Sep 10 12:32:19.575313 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:19.575270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h" event={"ID":"64fb7a24-54f4-4086-b4fd-f502603a60c7","Type":"ContainerStarted","Data":"106b9ab89b608ae53c5ca282c4bf8bd3eb0296d94f44fce4bd7ea03cf968a627"}
Sep 10 12:32:19.575799 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:19.575520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"
Sep 10 12:32:19.582396 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:19.582370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h"
Sep 10 12:32:19.595920 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:19.595866    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-svb9h" podStartSLOduration=2.158586201 podStartE2EDuration="3.595847595s" totalImagesPullingTime="1.437261394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:19.593127077 +0000 UTC m=+114.115296130" watchObservedRunningTime="2026-09-10 12:32:19.595847595 +0000 UTC m=+114.118016647"
Sep 10 12:32:20.061850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:20.061809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:20.130247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:20.130205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:20.964952 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:20.964916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:21.312285 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:21.312245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:21.414115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:21.414080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:22.064611 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:22.064560    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:22.133495 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:22.133460    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:22.950411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:22.950363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:23.014245 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:23.014215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:23.889884 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:23.889849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:24.689119 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:24.689086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:24.928939 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:24.928906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:24.993560 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:24.993486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:26.201147 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.201115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:26.307205 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.307174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:26.475210 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.475123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:26.534064 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.534018    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"]
Sep 10 12:32:26.538959 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.538936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.542667 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.542644    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dcqk4\""
Sep 10 12:32:26.542777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.542690    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 12:32:26.542777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.542643    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 12:32:26.542890 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.542647    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 12:32:26.543042 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.543023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 12:32:26.543151 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.543029    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 12:32:26.557051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.557002    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"]
Sep 10 12:32:26.566519 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.566499    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"]
Sep 10 12:32:26.569969 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.569948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.570430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.570387    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:26.572509 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.572490    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 12:32:26.572824 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.572808    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 12:32:26.574140 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.574121    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 12:32:26.574260 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.574240    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5pf58\""
Sep 10 12:32:26.582884 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.582860    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"]
Sep 10 12:32:26.593489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.593469    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dk8b2"]
Sep 10 12:32:26.596929 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.596897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.603067 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.603031    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 12:32:26.603288 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.603242    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 12:32:26.604965 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.604945    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 12:32:26.608900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.608881    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxq69\""
Sep 10 12:32:26.633531 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.633507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06744632-29d8-4339-a789-1d53398c153a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.633629 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.633543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.633629 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.633570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.633629 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.633622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lm6\" (UniqueName: \"kubernetes.io/projected/06744632-29d8-4339-a789-1d53398c153a-kube-api-access-k9lm6\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.690961 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.690939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:26.734985 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.734905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/70f7e891-ffa9-4196-a488-e84d0ff56f7a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.734985 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.734958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735194 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.735194 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lm6\" (UniqueName: \"kubernetes.io/projected/06744632-29d8-4339-a789-1d53398c153a-kube-api-access-k9lm6\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.735194 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1218ff9-94f8-493a-aef5-599b24fca4b9-metrics-client-ca\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735194 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqps\" (UniqueName: \"kubernetes.io/projected/f1218ff9-94f8-493a-aef5-599b24fca4b9-kube-api-access-ltqps\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735360 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-root\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735407 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735444 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9bk\" (UniqueName: \"kubernetes.io/projected/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-api-access-bs9bk\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.735485 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-wtmp\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735524 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.735565 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06744632-29d8-4339-a789-1d53398c153a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.735565 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.735665 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-sys\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735665 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.735742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-textfile\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f7e891-ffa9-4196-a488-e84d0ff56f7a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.735742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-tls\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.735858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.735749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.735897 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:32:26.735868    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 10 12:32:26.735946 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:32:26.735920    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-tls podName:06744632-29d8-4339-a789-1d53398c153a nodeName:}" failed. No retries permitted until 2026-09-10 12:32:27.23590242 +0000 UTC m=+121.758071452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-x5q99" (UID: "06744632-29d8-4339-a789-1d53398c153a") : secret "openshift-state-metrics-tls" not found
Sep 10 12:32:26.736337 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.736313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06744632-29d8-4339-a789-1d53398c153a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.738378 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.738354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.746328 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.746306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lm6\" (UniqueName: \"kubernetes.io/projected/06744632-29d8-4339-a789-1d53398c153a-kube-api-access-k9lm6\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:26.780759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.780729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:26.837005 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.836978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-textfile\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837163 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f7e891-ffa9-4196-a488-e84d0ff56f7a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.837163 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-tls\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837278 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/70f7e891-ffa9-4196-a488-e84d0ff56f7a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.837278 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837278 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.837489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1218ff9-94f8-493a-aef5-599b24fca4b9-metrics-client-ca\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqps\" (UniqueName: \"kubernetes.io/projected/f1218ff9-94f8-493a-aef5-599b24fca4b9-kube-api-access-ltqps\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-textfile\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-root\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9bk\" (UniqueName: \"kubernetes.io/projected/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-api-access-bs9bk\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.837489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-wtmp\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.837865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.837865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-sys\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/70f7e891-ffa9-4196-a488-e84d0ff56f7a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.837865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-root\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.837865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-sys\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.838139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.837916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1218ff9-94f8-493a-aef5-599b24fca4b9-metrics-client-ca\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.838139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.838084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.838139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.838128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-wtmp\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.838292 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.838218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f7e891-ffa9-4196-a488-e84d0ff56f7a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.838346 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.838301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.840096 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.840045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-tls\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.840504 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.840480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1218ff9-94f8-493a-aef5-599b24fca4b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.840708 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.840686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.841348 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.841318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.852330 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.852309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9bk\" (UniqueName: \"kubernetes.io/projected/70f7e891-ffa9-4196-a488-e84d0ff56f7a-kube-api-access-bs9bk\") pod \"kube-state-metrics-74cb57df7-5k5lq\" (UID: \"70f7e891-ffa9-4196-a488-e84d0ff56f7a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.852430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.852340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqps\" (UniqueName: \"kubernetes.io/projected/f1218ff9-94f8-493a-aef5-599b24fca4b9-kube-api-access-ltqps\") pod \"node-exporter-dk8b2\" (UID: \"f1218ff9-94f8-493a-aef5-599b24fca4b9\") " pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:26.884010 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.883937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"
Sep 10 12:32:26.885003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.884967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:26.907845 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:26.907822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dk8b2"
Sep 10 12:32:27.026427 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.026391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:27.127702 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.127671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:27.241905 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.241867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:27.244571 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.244543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06744632-29d8-4339-a789-1d53398c153a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-x5q99\" (UID: \"06744632-29d8-4339-a789-1d53398c153a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:27.449654 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.449542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"
Sep 10 12:32:27.478804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.478771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:27.553278 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.553246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:27.557296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.557272    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:32:27.561103 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.561084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.563748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.563725    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 12:32:27.563748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.563736    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 12:32:27.563748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.563746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 12:32:27.564105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.563736    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 12:32:27.564105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.563858    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 12:32:27.564105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.563990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 12:32:27.564341 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.564204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 12:32:27.564341 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.564247    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 12:32:27.564341 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.564246    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 12:32:27.564515 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.564470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pjmps\""
Sep 10 12:32:27.581907 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.581883    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:32:27.645638 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-config-volume\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.645784 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.645784 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.645784 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.645784 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-config-out\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.645784 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.645962 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbt5\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-kube-api-access-sbbt5\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.645962 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.645962 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.646060 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.645964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-web-config\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.646060 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.646002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.646060 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.646056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.646158 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.646088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.747302 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.747302 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.747302 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-web-config\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.747566 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.747566 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.747566 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-config-volume\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-config-out\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.747825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbt5\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-kube-api-access-sbbt5\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748457 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.748136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.748662 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:32:27.748635    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 10 12:32:27.748790 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:32:27.748719    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls podName:bda45807-387e-443d-bc39-c02b4fcd42b5 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:28.248697595 +0000 UTC m=+122.770866628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5") : secret "alertmanager-main-tls" not found
Sep 10 12:32:27.749148 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.749077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.750256 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.749720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.751126 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.750303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.751126 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.750760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-config-out\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.751126 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.751051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-config-volume\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.753040 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.751789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.753040 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.751895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.753040 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.751940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-web-config\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.753446 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.753423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.753864 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.753844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:27.757343 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:27.757321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbt5\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-kube-api-access-sbbt5\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:28.252913 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:28.252879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:28.255853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:28.255826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:28.272889 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:28.272846    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:28.472035 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:28.471996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:32:28.912839 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:28.912813    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77cf9fb475-gmzzh"]
Sep 10 12:32:28.916322 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:28.916300    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:28.928716 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:28.927138    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 12:32:28.935317 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:28.935293    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77cf9fb475-gmzzh"]
Sep 10 12:32:29.054364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.054329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq"]
Sep 10 12:32:29.056841 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.056813    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99"]
Sep 10 12:32:29.058816 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:29.058780    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f7e891_ffa9_4196_a488_e84d0ff56f7a.slice/crio-5b312d5f9bca9142ff15eb46a9fb3a90c4144cfcced2b03151bf13026201de17 WatchSource:0}: Error finding container 5b312d5f9bca9142ff15eb46a9fb3a90c4144cfcced2b03151bf13026201de17: Status 404 returned error can't find the container with id 5b312d5f9bca9142ff15eb46a9fb3a90c4144cfcced2b03151bf13026201de17
Sep 10 12:32:29.060407 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:29.060197    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06744632_29d8_4339_a789_1d53398c153a.slice/crio-2fe1c48c4457273f5b6841a0369d5b9b786ff800efc3cf8e064f7f12be61a521 WatchSource:0}: Error finding container 2fe1c48c4457273f5b6841a0369d5b9b786ff800efc3cf8e064f7f12be61a521: Status 404 returned error can't find the container with id 2fe1c48c4457273f5b6841a0369d5b9b786ff800efc3cf8e064f7f12be61a521
Sep 10 12:32:29.060581 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.060551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-trusted-ca-bundle\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.060716 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.060626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-oauth-config\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.060785 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.060724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-service-ca\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.060785 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.060758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-serving-cert\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.060893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.060795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrjr\" (UniqueName: \"kubernetes.io/projected/bee5151f-1d8e-4fab-bf05-72582b0287e8-kube-api-access-qvrjr\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.060893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.060836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-oauth-serving-cert\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.060893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.060861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-config\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.066647 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.066620    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:32:29.071677 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:29.071575    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda45807_387e_443d_bc39_c02b4fcd42b5.slice/crio-1910508b49f60c1d63a41122911c74a2d73726079c2abe5edb721293a4ec4886 WatchSource:0}: Error finding container 1910508b49f60c1d63a41122911c74a2d73726079c2abe5edb721293a4ec4886: Status 404 returned error can't find the container with id 1910508b49f60c1d63a41122911c74a2d73726079c2abe5edb721293a4ec4886
Sep 10 12:32:29.161374 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.161340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-service-ca\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.161518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.161389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-serving-cert\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.161518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.161424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrjr\" (UniqueName: \"kubernetes.io/projected/bee5151f-1d8e-4fab-bf05-72582b0287e8-kube-api-access-qvrjr\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.162658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.162085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-oauth-serving-cert\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.162658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.162136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-config\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.162658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.162172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-service-ca\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.162658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.162230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-trusted-ca-bundle\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.162658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.162257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-oauth-config\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.162984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.162765    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-oauth-serving-cert\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.163523 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.163488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-config\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.163637 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.163611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-trusted-ca-bundle\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.165710 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.165683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-oauth-config\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.166137 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.166115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-serving-cert\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.171480 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.171458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrjr\" (UniqueName: \"kubernetes.io/projected/bee5151f-1d8e-4fab-bf05-72582b0287e8-kube-api-access-qvrjr\") pod \"console-77cf9fb475-gmzzh\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") " pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.233222 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.233161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:29.469949 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.469903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:29.529822 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.529774    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77cf9fb475-gmzzh"]
Sep 10 12:32:29.533084 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:29.533049    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee5151f_1d8e_4fab_bf05_72582b0287e8.slice/crio-36901da0f09fdfbd3176f51ef8db87c40a0f86e99cfc2ca1cdf6af135eb6756c WatchSource:0}: Error finding container 36901da0f09fdfbd3176f51ef8db87c40a0f86e99cfc2ca1cdf6af135eb6756c: Status 404 returned error can't find the container with id 36901da0f09fdfbd3176f51ef8db87c40a0f86e99cfc2ca1cdf6af135eb6756c
Sep 10 12:32:29.533206 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.533150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:29.618282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.618170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-tmlbj" event={"ID":"fad3daa4-04a8-4bcb-a8a2-875b5262e073","Type":"ContainerStarted","Data":"7ebad5bb08d2377ff73d35bf4653985ac6b5834c171973f9cba297bb9c18a212"}
Sep 10 12:32:29.618282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.618242    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-tmlbj"
Sep 10 12:32:29.621637 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.621585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dk8b2" event={"ID":"f1218ff9-94f8-493a-aef5-599b24fca4b9","Type":"ContainerStarted","Data":"065a3b088e6a5fafcabc3d5363af8e86422f8f38c8e93752a4bc2d8a90686570"}
Sep 10 12:32:29.623481 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.623407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerStarted","Data":"1910508b49f60c1d63a41122911c74a2d73726079c2abe5edb721293a4ec4886"}
Sep 10 12:32:29.630750 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.630459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99" event={"ID":"06744632-29d8-4339-a789-1d53398c153a","Type":"ContainerStarted","Data":"ccd40937f8c8c673e63023a86fc97d0895eb4e71ec9550bb65f32c2d820e5087"}
Sep 10 12:32:29.630750 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.630521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99" event={"ID":"06744632-29d8-4339-a789-1d53398c153a","Type":"ContainerStarted","Data":"f4903fa7d5e2ba57f0b39969c06db39ef611067561487e8492cd3376a610ba67"}
Sep 10 12:32:29.630750 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.630550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99" event={"ID":"06744632-29d8-4339-a789-1d53398c153a","Type":"ContainerStarted","Data":"2fe1c48c4457273f5b6841a0369d5b9b786ff800efc3cf8e064f7f12be61a521"}
Sep 10 12:32:29.638470 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.638403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cf9fb475-gmzzh" event={"ID":"bee5151f-1d8e-4fab-bf05-72582b0287e8","Type":"ContainerStarted","Data":"36901da0f09fdfbd3176f51ef8db87c40a0f86e99cfc2ca1cdf6af135eb6756c"}
Sep 10 12:32:29.642113 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.641407    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-tmlbj" podStartSLOduration=1.890511108 podStartE2EDuration="19.641389429s" totalImagesPullingTime="17.750878321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:29.640747819 +0000 UTC m=+124.162916888" watchObservedRunningTime="2026-09-10 12:32:29.641389429 +0000 UTC m=+124.163558482"
Sep 10 12:32:29.647166 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.646670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-tmlbj"
Sep 10 12:32:29.647682 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.647630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6f59cdd-2w6d8" event={"ID":"f6169f56-1dc5-40a8-b629-7b6356066f8f","Type":"ContainerStarted","Data":"146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158"}
Sep 10 12:32:29.651123 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.651055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq" event={"ID":"70f7e891-ffa9-4196-a488-e84d0ff56f7a","Type":"ContainerStarted","Data":"5b312d5f9bca9142ff15eb46a9fb3a90c4144cfcced2b03151bf13026201de17"}
Sep 10 12:32:29.798718 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.798655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:29.902186 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:29.902101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:30.005315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.005223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:30.157915 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.157827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:30.235698 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.235632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:30.657543 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.657502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cf9fb475-gmzzh" event={"ID":"bee5151f-1d8e-4fab-bf05-72582b0287e8","Type":"ContainerStarted","Data":"44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008"}
Sep 10 12:32:30.681895 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.681835    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77cf9fb475-gmzzh" podStartSLOduration=2.68181624 podStartE2EDuration="2.68181624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:30.681105624 +0000 UTC m=+125.203274676" watchObservedRunningTime="2026-09-10 12:32:30.68181624 +0000 UTC m=+125.203985289"
Sep 10 12:32:30.683028 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.682983    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85c6f59cdd-2w6d8" podStartSLOduration=2.9214594 podStartE2EDuration="13.682970004s" totalImagesPullingTime="10.761510604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:29.689446675 +0000 UTC m=+124.211615727" watchObservedRunningTime="2026-09-10 12:32:30.682970004 +0000 UTC m=+125.205139057"
Sep 10 12:32:30.930869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.930798    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f74d6669d-4c55b"]
Sep 10 12:32:30.959111 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.959062    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f74d6669d-4c55b"]
Sep 10 12:32:30.959333 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.959195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:30.963350 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.963325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 12:32:30.963560 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.963346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6q3uatoi86art\""
Sep 10 12:32:30.963560 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.963356    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 12:32:30.963560 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.963390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 12:32:30.963560 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.963515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-22b89\""
Sep 10 12:32:30.963776 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:30.963738    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 12:32:31.085442 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.085410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-audit-log\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.085717 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.085460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-secret-metrics-server-client-certs\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.085717 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.085490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-client-ca-bundle\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.085717 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.085526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-metrics-server-audit-profiles\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.085717 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.085576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qrx\" (UniqueName: \"kubernetes.io/projected/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-kube-api-access-58qrx\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.085717 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.085650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.085717 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.085692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-secret-metrics-server-tls\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.186909 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.186822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-client-ca-bundle\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.186909 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.186880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-metrics-server-audit-profiles\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.187115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.186935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qrx\" (UniqueName: \"kubernetes.io/projected/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-kube-api-access-58qrx\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.187115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.186992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.187115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.187033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-secret-metrics-server-tls\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.187115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.187090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-audit-log\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.187303 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.187127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-secret-metrics-server-client-certs\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.187822 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.187749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-audit-log\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.188394 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.188063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-metrics-server-audit-profiles\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.188726 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.188669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.189974 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.189950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-client-ca-bundle\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.191199 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.191173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-secret-metrics-server-client-certs\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.191682 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.191641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-secret-metrics-server-tls\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.200055 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.200012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qrx\" (UniqueName: \"kubernetes.io/projected/eacad1b9-2965-4fb4-8310-6d5a1e78a0fe-kube-api-access-58qrx\") pod \"metrics-server-5f74d6669d-4c55b\" (UID: \"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe\") " pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.227282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.227244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:31.273968 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.273936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:31.327233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.327204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:31.467096 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.467034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:31.769045 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.767766    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b859c5658-rlzt4"]
Sep 10 12:32:31.794722 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.793131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b859c5658-rlzt4"]
Sep 10 12:32:31.794722 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.793270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.796090 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.796061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 12:32:31.799643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.797696    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-p2f97\""
Sep 10 12:32:31.799643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.797942    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 12:32:31.799643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.798333    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 12:32:31.799643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.798551    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 12:32:31.799643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.798895    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 12:32:31.805525 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.805412    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 12:32:31.894414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.894373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-serving-certs-ca-bundle\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.894743 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.894689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-telemeter-client-tls\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.894903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.894758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.894903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.894783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.894903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.894872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-metrics-client-ca\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.895150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.894979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-federate-client-tls\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.895150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.895054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txv4\" (UniqueName: \"kubernetes.io/projected/256405e5-57f3-4649-9d2b-1389fc955e9a-kube-api-access-5txv4\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.895150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.895107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-secret-telemeter-client\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.958567 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.958529    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f74d6669d-4c55b"]
Sep 10 12:32:31.995949 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.995873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-federate-client-tls\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.995949 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.995941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txv4\" (UniqueName: \"kubernetes.io/projected/256405e5-57f3-4649-9d2b-1389fc955e9a-kube-api-access-5txv4\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.996154 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.995974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-secret-telemeter-client\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.996154 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.996032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-serving-certs-ca-bundle\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.996154 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.996066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-telemeter-client-tls\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.996154 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.996116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.996154 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.996146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.996395 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.996200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-metrics-client-ca\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.997308 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.996977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-metrics-client-ca\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.997308 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.997127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-serving-certs-ca-bundle\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.998180 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.998132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/256405e5-57f3-4649-9d2b-1389fc955e9a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:31.999251 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:31.999216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-federate-client-tls\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:32.000144 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.000097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:32.001484 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.001011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-telemeter-client-tls\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:32.004191 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.004168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/256405e5-57f3-4649-9d2b-1389fc955e9a-secret-telemeter-client\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:32.006747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.006724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txv4\" (UniqueName: \"kubernetes.io/projected/256405e5-57f3-4649-9d2b-1389fc955e9a-kube-api-access-5txv4\") pod \"telemeter-client-b859c5658-rlzt4\" (UID: \"256405e5-57f3-4649-9d2b-1389fc955e9a\") " pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:32.006841 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.006822    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:32.107866 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.107835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4"
Sep 10 12:32:32.114280 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.114206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:32.449674 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.447659    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b859c5658-rlzt4"]
Sep 10 12:32:32.668183 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.668135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4" event={"ID":"256405e5-57f3-4649-9d2b-1389fc955e9a","Type":"ContainerStarted","Data":"2be936c76302f9c1be9b5ee54b38ea75db743f89fea707343faf4d24d255f03f"}
Sep 10 12:32:32.671003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.670974    2577 generic.go:356] "Generic (PLEG): container finished" podID="f1218ff9-94f8-493a-aef5-599b24fca4b9" containerID="cae3ecf74a64600b7900b443fcbc3fe03cedd12b38b016bb6a8dd5e63b311f70" exitCode=0
Sep 10 12:32:32.671233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.671095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dk8b2" event={"ID":"f1218ff9-94f8-493a-aef5-599b24fca4b9","Type":"ContainerDied","Data":"cae3ecf74a64600b7900b443fcbc3fe03cedd12b38b016bb6a8dd5e63b311f70"}
Sep 10 12:32:32.673925 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.673900    2577 generic.go:356] "Generic (PLEG): container finished" podID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerID="43ecc4b81325e1ebda8a57841b4c6f40e61a3cd197839e35dfb3281652f11b4a" exitCode=0
Sep 10 12:32:32.674031 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.674012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerDied","Data":"43ecc4b81325e1ebda8a57841b4c6f40e61a3cd197839e35dfb3281652f11b4a"}
Sep 10 12:32:32.678854 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.678510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99" event={"ID":"06744632-29d8-4339-a789-1d53398c153a","Type":"ContainerStarted","Data":"1a5104380f57f4f04c79a73e40e58c8569408f3b4279247173c0c05e173c9a22"}
Sep 10 12:32:32.689114 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.685212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b" event={"ID":"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe","Type":"ContainerStarted","Data":"fe39318e368b79998d0ed3649a52531c8921f7ab1143a51e5b9073a7933e0cd2"}
Sep 10 12:32:32.689815 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.689789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq" event={"ID":"70f7e891-ffa9-4196-a488-e84d0ff56f7a","Type":"ContainerStarted","Data":"c5de649b9efd0189f6d7f300530234516711b4a12218dd16f76b55ca16e77d0c"}
Sep 10 12:32:32.689915 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.689861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq" event={"ID":"70f7e891-ffa9-4196-a488-e84d0ff56f7a","Type":"ContainerStarted","Data":"37e6fcca13531415e53537faeba5665aade7c8227d7d6a1a4a47711b06b31495"}
Sep 10 12:32:32.689915 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.689903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq" event={"ID":"70f7e891-ffa9-4196-a488-e84d0ff56f7a","Type":"ContainerStarted","Data":"9eb3b3fc1d02dee39e1d95ea595b62dbf0685c10ff872a26dea6b862307d7a94"}
Sep 10 12:32:32.713011 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.712961    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-x5q99" podStartSLOduration=3.447349855 podStartE2EDuration="6.712943205s" totalImagesPullingTime="3.26559335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:32.711001258 +0000 UTC m=+127.233170311" watchObservedRunningTime="2026-09-10 12:32:32.712943205 +0000 UTC m=+127.235112259"
Sep 10 12:32:32.760572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.760485    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:32.761276 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.761221    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:32:32.768136 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.768091    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-5k5lq" podStartSLOduration=4.146524923 podStartE2EDuration="6.76807297s" totalImagesPullingTime="2.621548047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:32.766162395 +0000 UTC m=+127.288331470" watchObservedRunningTime="2026-09-10 12:32:32.76807297 +0000 UTC m=+127.290242020"
Sep 10 12:32:32.793086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.793048    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:32:32.793588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.793269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.796644 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.796625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.797487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.797567    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.796976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.797093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.797277    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xgnxq\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.797222    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.797052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.797491    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 12:32:32.798034 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.798000    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 12:32:32.798512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.798168    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2g0inaf4ilrs7\""
Sep 10 12:32:32.798512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.798273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 12:32:32.798512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.798422    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 12:32:32.798512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.798467    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 12:32:32.799020 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.798793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 12:32:32.802629 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.802392    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 12:32:32.906375 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906478 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906478 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906478 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjll\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-kube-api-access-gqjll\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906766 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906766 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906766 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906872 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906872 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.906964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.907106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.906977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.907106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.907012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.907106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.907054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.907106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.907081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:32.907268 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:32.907123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.007903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.007859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008070 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.007914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008070 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.007953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008070 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008070 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008070 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008326 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjll\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-kube-api-access-gqjll\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.008770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.009063 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.008874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.009149 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.009122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.009747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.009381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.011984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.011105    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.011984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.011629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.011984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.011738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.012430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.012403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.013037 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.013012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.013120 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.013043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.013185 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.013120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.014054 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.013495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.015366 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.014856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.015366 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.014884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.017623 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.017406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.017623 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.017544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.018225 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.018184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.018706 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.018663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.020537 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.020510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.021544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.021513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.023330 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.023309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjll\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-kube-api-access-gqjll\") pod \"prometheus-k8s-0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.109231 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.109167    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:33.290193 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.290123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:33.391179 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.391065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:33.449133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.448998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:32:33.500289 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.500252    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:33.705330 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.704489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dk8b2" event={"ID":"f1218ff9-94f8-493a-aef5-599b24fca4b9","Type":"ContainerStarted","Data":"2774b74796723f0433aeb4b21c17455d5ee5072bb8e3407f7d33b5fa08fced85"}
Sep 10 12:32:33.705330 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.704564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dk8b2" event={"ID":"f1218ff9-94f8-493a-aef5-599b24fca4b9","Type":"ContainerStarted","Data":"ec49e38eaf26efd4a9253677c9663ef77c2ae09728606dc89777e8030986efae"}
Sep 10 12:32:33.741265 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.741209    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dk8b2" podStartSLOduration=4.704181757 podStartE2EDuration="7.74118799s" totalImagesPullingTime="2.907338309s" totalInitContainerRuntime="129.667924ms" isStatefulPod=true podCreationTimestamp="2026-09-10 12:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:33.739398826 +0000 UTC m=+128.261567880" watchObservedRunningTime="2026-09-10 12:32:33.74118799 +0000 UTC m=+128.263357044"
Sep 10 12:32:33.861816 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:33.861742    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:34.229147 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:34.229103    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcb3d67_2dc8_4141_982d_f315ccc023a0.slice/crio-ce6c4a37e51e4772911e0e17c963d26783d08e5ce4c255ef97e944140c9c7e28 WatchSource:0}: Error finding container ce6c4a37e51e4772911e0e17c963d26783d08e5ce4c255ef97e944140c9c7e28: Status 404 returned error can't find the container with id ce6c4a37e51e4772911e0e17c963d26783d08e5ce4c255ef97e944140c9c7e28
Sep 10 12:32:34.710931 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:34.710889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerStarted","Data":"ce6c4a37e51e4772911e0e17c963d26783d08e5ce4c255ef97e944140c9c7e28"}
Sep 10 12:32:34.827960 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:34.827920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:32:34.830500 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:34.830472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb054c1-3406-4776-b4b6-b44b6d1f9849-metrics-certs\") pod \"network-metrics-daemon-dj27m\" (UID: \"2eb054c1-3406-4776-b4b6-b44b6d1f9849\") " pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:32:35.107741 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:35.107705    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sz48d\""
Sep 10 12:32:35.115377 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:35.115349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj27m"
Sep 10 12:32:36.280368 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.279261    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dj27m"]
Sep 10 12:32:36.284326 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:32:36.284279    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb054c1_3406_4776_b4b6_b44b6d1f9849.slice/crio-1c84032880166effdac3e187068639dfe5b63d8c50f79eae071151039f3f600f WatchSource:0}: Error finding container 1c84032880166effdac3e187068639dfe5b63d8c50f79eae071151039f3f600f: Status 404 returned error can't find the container with id 1c84032880166effdac3e187068639dfe5b63d8c50f79eae071151039f3f600f
Sep 10 12:32:36.723523 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.723488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4" event={"ID":"256405e5-57f3-4649-9d2b-1389fc955e9a","Type":"ContainerStarted","Data":"bd214755ec86fa34dfa8f7705b24963480e1d407b3d7d19142019157c5b09de5"}
Sep 10 12:32:36.723671 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.723611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4" event={"ID":"256405e5-57f3-4649-9d2b-1389fc955e9a","Type":"ContainerStarted","Data":"1edab3cb3278bf05bd9f81061502ef88e073e55040b597ebf7ba3a83d1da8a62"}
Sep 10 12:32:36.723671 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.723648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4" event={"ID":"256405e5-57f3-4649-9d2b-1389fc955e9a","Type":"ContainerStarted","Data":"63e9cea2889e0067f87a966bb631f5fc2928d4fa20f5ea3a38a351915b374507"}
Sep 10 12:32:36.727123 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.727095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerStarted","Data":"b1fa3dface24abd62c5579492ba7527b9fc98786f1be14cc38e47ee73c5c60a1"}
Sep 10 12:32:36.727227 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.727167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerStarted","Data":"c4194156ca956d6beb6f607a83349b7f074efa6e14ff76146602509b325bd439"}
Sep 10 12:32:36.727227 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.727212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerStarted","Data":"a5ae41b5d30f034c28b1648d24c98e82a444574c9db133697d77f7e8f6de2d62"}
Sep 10 12:32:36.727337 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.727253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerStarted","Data":"366dd0baed40f43d4b53e61d3194feebbcb4e9cd5508eae1069fe0d27e2e9f54"}
Sep 10 12:32:36.728974 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.728945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b" event={"ID":"eacad1b9-2965-4fb4-8310-6d5a1e78a0fe","Type":"ContainerStarted","Data":"31eb93934a587acc26af22c5e6578f998025a3b9078ae102123aabeac8b18472"}
Sep 10 12:32:36.730985 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.730958    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerID="347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a" exitCode=0
Sep 10 12:32:36.731081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.731039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerDied","Data":"347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a"}
Sep 10 12:32:36.732441 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.732419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dj27m" event={"ID":"2eb054c1-3406-4776-b4b6-b44b6d1f9849","Type":"ContainerStarted","Data":"1c84032880166effdac3e187068639dfe5b63d8c50f79eae071151039f3f600f"}
Sep 10 12:32:36.751385 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.751333    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b859c5658-rlzt4" podStartSLOduration=2.23402739 podStartE2EDuration="5.751315794s" totalImagesPullingTime="3.517288404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:36.749564894 +0000 UTC m=+131.271733946" watchObservedRunningTime="2026-09-10 12:32:36.751315794 +0000 UTC m=+131.273484848"
Sep 10 12:32:36.816658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:36.816585    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b" podStartSLOduration=2.882917108 podStartE2EDuration="6.816563933s" totalImagesPullingTime="3.933646825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:36.815640008 +0000 UTC m=+131.337809085" watchObservedRunningTime="2026-09-10 12:32:36.816563933 +0000 UTC m=+131.338732983"
Sep 10 12:32:37.527360 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:37.527320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:37.540910 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:37.540868    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c6f59cdd-2w6d8"]
Sep 10 12:32:37.604203 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:37.604171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:37.727907 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:37.727841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:32:37.743047 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:37.743008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerStarted","Data":"43421f0f724251fb700a22a56122004c81db94696c96e7e128bc754e3c3a8139"}
Sep 10 12:32:38.413241 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:38.413216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:38.512923 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:38.512895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:38.748132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:38.748108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dj27m" event={"ID":"2eb054c1-3406-4776-b4b6-b44b6d1f9849","Type":"ContainerStarted","Data":"6d94343122add84fe05ca616bb1cf83eae4f5f41bbe06dec647fd98e01c482de"}
Sep 10 12:32:38.748406 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:38.748193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dj27m" event={"ID":"2eb054c1-3406-4776-b4b6-b44b6d1f9849","Type":"ContainerStarted","Data":"2d5eb2c34644e0f0419b918cd98a32eb9ccf214b4a1d5280b33d274eb38a3b19"}
Sep 10 12:32:38.771299 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:38.771239    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dj27m" podStartSLOduration=131.127402818 podStartE2EDuration="2m12.771218956s" totalImagesPullingTime="1.643816138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:38.770551805 +0000 UTC m=+133.292720861" watchObservedRunningTime="2026-09-10 12:32:38.771218956 +0000 UTC m=+133.293388008"
Sep 10 12:32:38.916128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:38.916066    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:38.977721 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:38.977639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:39.233711 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:39.233619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:39.233711 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:39.233696    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:39.240544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:39.240509    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:39.757407 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:39.757354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerStarted","Data":"a4ab22e0e385136b15e3362829d0b7866b9e7fc99b85a5ef343334db92b17c54"}
Sep 10 12:32:39.761576 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:39.761553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:32:39.796649 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:39.796561    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.977168474 podStartE2EDuration="12.796546799s" totalImagesPullingTime="7.812659223s" totalInitContainerRuntime="6.719102ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:39.794196445 +0000 UTC m=+134.316365509" watchObservedRunningTime="2026-09-10 12:32:39.796546799 +0000 UTC m=+134.318715863"
Sep 10 12:32:40.763342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:40.763312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerStarted","Data":"14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4"}
Sep 10 12:32:40.763697 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:40.763378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerStarted","Data":"4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33"}
Sep 10 12:32:42.740547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:42.740510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:42.810502 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:42.810475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:43.780077 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:43.780041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerStarted","Data":"0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b"}
Sep 10 12:32:43.780564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:43.780103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerStarted","Data":"83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c"}
Sep 10 12:32:43.780564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:43.780130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerStarted","Data":"f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7"}
Sep 10 12:32:43.780564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:43.780154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerStarted","Data":"f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be"}
Sep 10 12:32:43.837587 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:43.837539    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.920144935 podStartE2EDuration="11.837522806s" totalImagesPullingTime="5.900600025s" totalInitContainerRuntime="16.777846ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:43.834732596 +0000 UTC m=+138.356901667" watchObservedRunningTime="2026-09-10 12:32:43.837522806 +0000 UTC m=+138.359691857"
Sep 10 12:32:44.468491 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:44.468452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:46.913643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:46.913606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:48.108226 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:48.108190    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77cf9fb475-gmzzh"]
Sep 10 12:32:48.109865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:48.109838    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:32:48.142993 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:48.142959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:48.240649 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:48.240586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:49.115549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:49.115516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:49.215470 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:49.215442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:50.114789 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:50.114754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:50.181558 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:50.181527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:51.274372 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:51.274343    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:51.274744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:51.274401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:32:53.148745 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:53.148717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:32:53.248885 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:32:53.248850    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:02.017004 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.016969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:02.114748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.114723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:02.215006 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.214978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:02.567134 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.567103    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85c6f59cdd-2w6d8" podUID="f6169f56-1dc5-40a8-b629-7b6356066f8f" containerName="console" containerID="cri-o://146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158" gracePeriod=15
Sep 10 12:33:02.865482 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.865449    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:33:02.868581 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.868547    2577 generic.go:356] "Generic (PLEG): container finished" podID="f6169f56-1dc5-40a8-b629-7b6356066f8f" containerID="146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158" exitCode=0
Sep 10 12:33:02.868708 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.868607    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6f59cdd-2w6d8"
Sep 10 12:33:02.868708 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.868581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6f59cdd-2w6d8" event={"ID":"f6169f56-1dc5-40a8-b629-7b6356066f8f","Type":"ContainerDied","Data":"146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158"}
Sep 10 12:33:02.868800 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.868730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6f59cdd-2w6d8" event={"ID":"f6169f56-1dc5-40a8-b629-7b6356066f8f","Type":"ContainerDied","Data":"69f8aa822aeb9c04e3f6563dfdcc09fef09c24d923cdd47c3e4356c7062e9909"}
Sep 10 12:33:02.868800 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.868760    2577 scope.go:117] "RemoveContainer" containerID="146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158"
Sep 10 12:33:02.876457 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.876441    2577 scope.go:117] "RemoveContainer" containerID="146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158"
Sep 10 12:33:02.876763 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:02.876734    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158\": container with ID starting with 146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158 not found: ID does not exist" containerID="146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158"
Sep 10 12:33:02.876844 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.876774    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158"} err="failed to get container status \"146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158\": rpc error: code = NotFound desc = could not find container \"146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158\": container with ID starting with 146efbe1dde831eb06eed29f04bf96b785a28bd9394568b436fb29a37cb8d158 not found: ID does not exist"
Sep 10 12:33:02.959459 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.959431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-service-ca\") pod \"f6169f56-1dc5-40a8-b629-7b6356066f8f\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") "
Sep 10 12:33:02.959622 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.959475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj4g8\" (UniqueName: \"kubernetes.io/projected/f6169f56-1dc5-40a8-b629-7b6356066f8f-kube-api-access-sj4g8\") pod \"f6169f56-1dc5-40a8-b629-7b6356066f8f\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") "
Sep 10 12:33:02.959622 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.959517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-oauth-serving-cert\") pod \"f6169f56-1dc5-40a8-b629-7b6356066f8f\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") "
Sep 10 12:33:02.959622 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.959572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-oauth-config\") pod \"f6169f56-1dc5-40a8-b629-7b6356066f8f\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") "
Sep 10 12:33:02.959796 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.959641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-serving-cert\") pod \"f6169f56-1dc5-40a8-b629-7b6356066f8f\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") "
Sep 10 12:33:02.959796 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.959699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-config\") pod \"f6169f56-1dc5-40a8-b629-7b6356066f8f\" (UID: \"f6169f56-1dc5-40a8-b629-7b6356066f8f\") "
Sep 10 12:33:02.959894 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.959821    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-service-ca" (OuterVolumeSpecName: "service-ca") pod "f6169f56-1dc5-40a8-b629-7b6356066f8f" (UID: "f6169f56-1dc5-40a8-b629-7b6356066f8f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:02.960029 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.960002    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:02.960145 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.960053    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f6169f56-1dc5-40a8-b629-7b6356066f8f" (UID: "f6169f56-1dc5-40a8-b629-7b6356066f8f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:02.960145 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.960077    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-config" (OuterVolumeSpecName: "console-config") pod "f6169f56-1dc5-40a8-b629-7b6356066f8f" (UID: "f6169f56-1dc5-40a8-b629-7b6356066f8f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:02.961858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.961831    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6169f56-1dc5-40a8-b629-7b6356066f8f-kube-api-access-sj4g8" (OuterVolumeSpecName: "kube-api-access-sj4g8") pod "f6169f56-1dc5-40a8-b629-7b6356066f8f" (UID: "f6169f56-1dc5-40a8-b629-7b6356066f8f"). InnerVolumeSpecName "kube-api-access-sj4g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:02.961858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.961837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f6169f56-1dc5-40a8-b629-7b6356066f8f" (UID: "f6169f56-1dc5-40a8-b629-7b6356066f8f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:02.961974 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:02.961877    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f6169f56-1dc5-40a8-b629-7b6356066f8f" (UID: "f6169f56-1dc5-40a8-b629-7b6356066f8f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:03.061222 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:03.061197    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-config\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:03.061222 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:03.061219    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj4g8\" (UniqueName: \"kubernetes.io/projected/f6169f56-1dc5-40a8-b629-7b6356066f8f-kube-api-access-sj4g8\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:03.061564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:03.061228    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6169f56-1dc5-40a8-b629-7b6356066f8f-oauth-serving-cert\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:03.061564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:03.061237    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-oauth-config\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:03.061564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:03.061246    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6169f56-1dc5-40a8-b629-7b6356066f8f-console-serving-cert\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:03.191781 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:03.191757    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c6f59cdd-2w6d8"]
Sep 10 12:33:03.195300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:03.195278    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c6f59cdd-2w6d8"]
Sep 10 12:33:03.999564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:03.999530    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6169f56-1dc5-40a8-b629-7b6356066f8f" path="/var/lib/kubelet/pods/f6169f56-1dc5-40a8-b629-7b6356066f8f/volumes"
Sep 10 12:33:08.147248 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:08.147202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:08.451498 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:08.451395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:11.280397 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:11.280365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:33:11.284235 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:11.284211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f74d6669d-4c55b"
Sep 10 12:33:13.132779 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.132730    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77cf9fb475-gmzzh" podUID="bee5151f-1d8e-4fab-bf05-72582b0287e8" containerName="console" containerID="cri-o://44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008" gracePeriod=15
Sep 10 12:33:13.408982 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.408930    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:33:13.553490 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553453    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-service-ca\") pod \"bee5151f-1d8e-4fab-bf05-72582b0287e8\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") "
Sep 10 12:33:13.553664 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-oauth-serving-cert\") pod \"bee5151f-1d8e-4fab-bf05-72582b0287e8\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") "
Sep 10 12:33:13.553664 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-trusted-ca-bundle\") pod \"bee5151f-1d8e-4fab-bf05-72582b0287e8\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") "
Sep 10 12:33:13.553664 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-serving-cert\") pod \"bee5151f-1d8e-4fab-bf05-72582b0287e8\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") "
Sep 10 12:33:13.553833 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-oauth-config\") pod \"bee5151f-1d8e-4fab-bf05-72582b0287e8\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") "
Sep 10 12:33:13.553833 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553719    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-config\") pod \"bee5151f-1d8e-4fab-bf05-72582b0287e8\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") "
Sep 10 12:33:13.553833 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvrjr\" (UniqueName: \"kubernetes.io/projected/bee5151f-1d8e-4fab-bf05-72582b0287e8-kube-api-access-qvrjr\") pod \"bee5151f-1d8e-4fab-bf05-72582b0287e8\" (UID: \"bee5151f-1d8e-4fab-bf05-72582b0287e8\") "
Sep 10 12:33:13.553995 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553865    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-service-ca" (OuterVolumeSpecName: "service-ca") pod "bee5151f-1d8e-4fab-bf05-72582b0287e8" (UID: "bee5151f-1d8e-4fab-bf05-72582b0287e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:13.553995 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.553961    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bee5151f-1d8e-4fab-bf05-72582b0287e8" (UID: "bee5151f-1d8e-4fab-bf05-72582b0287e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:13.554103 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.554044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bee5151f-1d8e-4fab-bf05-72582b0287e8" (UID: "bee5151f-1d8e-4fab-bf05-72582b0287e8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:13.554103 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.554090    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:13.554201 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.554108    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-oauth-serving-cert\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:13.554257 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.554243    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-config" (OuterVolumeSpecName: "console-config") pod "bee5151f-1d8e-4fab-bf05-72582b0287e8" (UID: "bee5151f-1d8e-4fab-bf05-72582b0287e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:13.556127 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.556092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee5151f-1d8e-4fab-bf05-72582b0287e8-kube-api-access-qvrjr" (OuterVolumeSpecName: "kube-api-access-qvrjr") pod "bee5151f-1d8e-4fab-bf05-72582b0287e8" (UID: "bee5151f-1d8e-4fab-bf05-72582b0287e8"). InnerVolumeSpecName "kube-api-access-qvrjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:13.556259 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.556170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bee5151f-1d8e-4fab-bf05-72582b0287e8" (UID: "bee5151f-1d8e-4fab-bf05-72582b0287e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:13.556259 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.556199    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bee5151f-1d8e-4fab-bf05-72582b0287e8" (UID: "bee5151f-1d8e-4fab-bf05-72582b0287e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:13.654770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.654742    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-serving-cert\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:13.654770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.654765    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-oauth-config\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:13.654770 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.654774    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-console-config\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:13.654949 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.654783    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvrjr\" (UniqueName: \"kubernetes.io/projected/bee5151f-1d8e-4fab-bf05-72582b0287e8-kube-api-access-qvrjr\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:13.654949 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.654792    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee5151f-1d8e-4fab-bf05-72582b0287e8-trusted-ca-bundle\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:13.914677 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.914642    2577 generic.go:356] "Generic (PLEG): container finished" podID="bee5151f-1d8e-4fab-bf05-72582b0287e8" containerID="44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008" exitCode=0
Sep 10 12:33:13.914827 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.914696    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cf9fb475-gmzzh"
Sep 10 12:33:13.914827 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.914726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cf9fb475-gmzzh" event={"ID":"bee5151f-1d8e-4fab-bf05-72582b0287e8","Type":"ContainerDied","Data":"44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008"}
Sep 10 12:33:13.914827 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.914781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cf9fb475-gmzzh" event={"ID":"bee5151f-1d8e-4fab-bf05-72582b0287e8","Type":"ContainerDied","Data":"36901da0f09fdfbd3176f51ef8db87c40a0f86e99cfc2ca1cdf6af135eb6756c"}
Sep 10 12:33:13.914827 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.914803    2577 scope.go:117] "RemoveContainer" containerID="44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008"
Sep 10 12:33:13.924126 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.924110    2577 scope.go:117] "RemoveContainer" containerID="44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008"
Sep 10 12:33:13.924389 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:13.924369    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008\": container with ID starting with 44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008 not found: ID does not exist" containerID="44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008"
Sep 10 12:33:13.924438 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.924398    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008"} err="failed to get container status \"44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008\": rpc error: code = NotFound desc = could not find container \"44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008\": container with ID starting with 44af958937f8a1d2d53f31552796a9fb63e6c80085b43e6387b283eaa9af0008 not found: ID does not exist"
Sep 10 12:33:13.936756 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.936731    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77cf9fb475-gmzzh"]
Sep 10 12:33:13.940716 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.940695    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77cf9fb475-gmzzh"]
Sep 10 12:33:13.998608 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:13.998575    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee5151f-1d8e-4fab-bf05-72582b0287e8" path="/var/lib/kubelet/pods/bee5151f-1d8e-4fab-bf05-72582b0287e8/volumes"
Sep 10 12:33:15.216789 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:15.216758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:15.827473 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:15.827435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:15.892693 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:15.892662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:21.238074 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:21.238004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:21.317528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:21.317499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:33.109635 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:33.109583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:33.127197 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:33.127175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:34.019648 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:34.019618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:42.656957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:42.656921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:42.757114 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:42.757077    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:46.941032 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:46.940991    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:33:46.941412 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:46.941283    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="alertmanager" containerID="cri-o://366dd0baed40f43d4b53e61d3194feebbcb4e9cd5508eae1069fe0d27e2e9f54" gracePeriod=120
Sep 10 12:33:46.941412 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:46.941359    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy-metric" containerID="cri-o://43421f0f724251fb700a22a56122004c81db94696c96e7e128bc754e3c3a8139" gracePeriod=120
Sep 10 12:33:46.941527 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:46.941393    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy-web" containerID="cri-o://c4194156ca956d6beb6f607a83349b7f074efa6e14ff76146602509b325bd439" gracePeriod=120
Sep 10 12:33:46.941527 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:46.941411    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="prom-label-proxy" containerID="cri-o://a4ab22e0e385136b15e3362829d0b7866b9e7fc99b85a5ef343334db92b17c54" gracePeriod=120
Sep 10 12:33:46.941527 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:46.941447    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy" containerID="cri-o://b1fa3dface24abd62c5579492ba7527b9fc98786f1be14cc38e47ee73c5c60a1" gracePeriod=120
Sep 10 12:33:46.941527 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:46.941425    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="config-reloader" containerID="cri-o://a5ae41b5d30f034c28b1648d24c98e82a444574c9db133697d77f7e8f6de2d62" gracePeriod=120
Sep 10 12:33:47.068496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:47.068470    2577 generic.go:356] "Generic (PLEG): container finished" podID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerID="a4ab22e0e385136b15e3362829d0b7866b9e7fc99b85a5ef343334db92b17c54" exitCode=0
Sep 10 12:33:47.068496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:47.068493    2577 generic.go:356] "Generic (PLEG): container finished" podID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerID="b1fa3dface24abd62c5579492ba7527b9fc98786f1be14cc38e47ee73c5c60a1" exitCode=0
Sep 10 12:33:47.068496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:47.068500    2577 generic.go:356] "Generic (PLEG): container finished" podID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerID="a5ae41b5d30f034c28b1648d24c98e82a444574c9db133697d77f7e8f6de2d62" exitCode=0
Sep 10 12:33:47.068680 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:47.068506    2577 generic.go:356] "Generic (PLEG): container finished" podID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerID="366dd0baed40f43d4b53e61d3194feebbcb4e9cd5508eae1069fe0d27e2e9f54" exitCode=0
Sep 10 12:33:47.068680 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:47.068531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerDied","Data":"a4ab22e0e385136b15e3362829d0b7866b9e7fc99b85a5ef343334db92b17c54"}
Sep 10 12:33:47.068680 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:47.068583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerDied","Data":"b1fa3dface24abd62c5579492ba7527b9fc98786f1be14cc38e47ee73c5c60a1"}
Sep 10 12:33:47.068680 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:47.068630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerDied","Data":"a5ae41b5d30f034c28b1648d24c98e82a444574c9db133697d77f7e8f6de2d62"}
Sep 10 12:33:47.068680 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:47.068654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerDied","Data":"366dd0baed40f43d4b53e61d3194feebbcb4e9cd5508eae1069fe0d27e2e9f54"}
Sep 10 12:33:48.077011 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.076983    2577 generic.go:356] "Generic (PLEG): container finished" podID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerID="43421f0f724251fb700a22a56122004c81db94696c96e7e128bc754e3c3a8139" exitCode=0
Sep 10 12:33:48.077011 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.077007    2577 generic.go:356] "Generic (PLEG): container finished" podID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerID="c4194156ca956d6beb6f607a83349b7f074efa6e14ff76146602509b325bd439" exitCode=0
Sep 10 12:33:48.077363 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.077050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerDied","Data":"43421f0f724251fb700a22a56122004c81db94696c96e7e128bc754e3c3a8139"}
Sep 10 12:33:48.077363 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.077111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerDied","Data":"c4194156ca956d6beb6f607a83349b7f074efa6e14ff76146602509b325bd439"}
Sep 10 12:33:48.177271 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.177245    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:48.238956 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.238933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-tls-assets\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.239108 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.238975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.239108 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.238997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239397    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-cluster-tls-config\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-main-db\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-metrics-client-ca\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-config-volume\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239734    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbbt5\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-kube-api-access-sbbt5\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239797    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-trusted-ca-bundle\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239849    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-web-config\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239926    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-config-out\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.240246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.239953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-web\") pod \"bda45807-387e-443d-bc39-c02b4fcd42b5\" (UID: \"bda45807-387e-443d-bc39-c02b4fcd42b5\") "
Sep 10 12:33:48.242404 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.241909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:48.242404 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.242335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:48.242720 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.242676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:48.242820 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.242794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:48.243724 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.243697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:48.245161 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.245130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-kube-api-access-sbbt5" (OuterVolumeSpecName: "kube-api-access-sbbt5") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "kube-api-access-sbbt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:48.245294 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.245269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-config-out" (OuterVolumeSpecName: "config-out") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:48.245545 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.245511    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:48.245762 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.245713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:48.245762 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.245732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:48.245912 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.245875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-config-volume" (OuterVolumeSpecName: "config-volume") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:48.249351 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.249248    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:48.257307 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.257277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-web-config" (OuterVolumeSpecName: "web-config") pod "bda45807-387e-443d-bc39-c02b4fcd42b5" (UID: "bda45807-387e-443d-bc39-c02b4fcd42b5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:48.340942 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.340909    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-main-db\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.340942 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.340937    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-metrics-client-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.340950    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.340963    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbbt5\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-kube-api-access-sbbt5\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.340976    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bda45807-387e-443d-bc39-c02b4fcd42b5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.340987    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-web-config\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.340998    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bda45807-387e-443d-bc39-c02b4fcd42b5-config-out\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.341010    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.341021    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bda45807-387e-443d-bc39-c02b4fcd42b5-tls-assets\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.341033    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.341045    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.341059    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-secret-alertmanager-main-tls\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.341139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:48.341070    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bda45807-387e-443d-bc39-c02b4fcd42b5-cluster-tls-config\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:49.084432 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.084398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bda45807-387e-443d-bc39-c02b4fcd42b5","Type":"ContainerDied","Data":"1910508b49f60c1d63a41122911c74a2d73726079c2abe5edb721293a4ec4886"}
Sep 10 12:33:49.084735 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.084458    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.085324 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.084490    2577 scope.go:117] "RemoveContainer" containerID="a4ab22e0e385136b15e3362829d0b7866b9e7fc99b85a5ef343334db92b17c54"
Sep 10 12:33:49.093869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.093852    2577 scope.go:117] "RemoveContainer" containerID="43421f0f724251fb700a22a56122004c81db94696c96e7e128bc754e3c3a8139"
Sep 10 12:33:49.100750 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.100735    2577 scope.go:117] "RemoveContainer" containerID="b1fa3dface24abd62c5579492ba7527b9fc98786f1be14cc38e47ee73c5c60a1"
Sep 10 12:33:49.108040 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.107741    2577 scope.go:117] "RemoveContainer" containerID="c4194156ca956d6beb6f607a83349b7f074efa6e14ff76146602509b325bd439"
Sep 10 12:33:49.109729 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.109704    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:33:49.114974 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.114954    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:33:49.115384 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.115368    2577 scope.go:117] "RemoveContainer" containerID="a5ae41b5d30f034c28b1648d24c98e82a444574c9db133697d77f7e8f6de2d62"
Sep 10 12:33:49.122389 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.122370    2577 scope.go:117] "RemoveContainer" containerID="366dd0baed40f43d4b53e61d3194feebbcb4e9cd5508eae1069fe0d27e2e9f54"
Sep 10 12:33:49.129161 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.129147    2577 scope.go:117] "RemoveContainer" containerID="43ecc4b81325e1ebda8a57841b4c6f40e61a3cd197839e35dfb3281652f11b4a"
Sep 10 12:33:49.999435 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:49.999402    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" path="/var/lib/kubelet/pods/bda45807-387e-443d-bc39-c02b4fcd42b5/volumes"
Sep 10 12:33:51.210586 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:51.210548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:33:51.211065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:51.210932    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="prometheus" containerID="cri-o://4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33" gracePeriod=600
Sep 10 12:33:51.211065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:51.210955    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="thanos-sidecar" containerID="cri-o://f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be" gracePeriod=600
Sep 10 12:33:51.211065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:51.210980    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="config-reloader" containerID="cri-o://14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4" gracePeriod=600
Sep 10 12:33:51.211065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:51.211005    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy-web" containerID="cri-o://f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7" gracePeriod=600
Sep 10 12:33:51.211065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:51.211004    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b" gracePeriod=600
Sep 10 12:33:51.211065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:51.210939    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy" containerID="cri-o://83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c" gracePeriod=600
Sep 10 12:33:52.100970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.100936    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerID="0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b" exitCode=0
Sep 10 12:33:52.100970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.100959    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerID="83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c" exitCode=0
Sep 10 12:33:52.100970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.100966    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerID="f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be" exitCode=0
Sep 10 12:33:52.100970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.100972    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerID="14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4" exitCode=0
Sep 10 12:33:52.100970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.100980    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerID="4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33" exitCode=0
Sep 10 12:33:52.101258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.101006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerDied","Data":"0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b"}
Sep 10 12:33:52.101258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.101067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerDied","Data":"83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c"}
Sep 10 12:33:52.101258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.101094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerDied","Data":"f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be"}
Sep 10 12:33:52.101258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.101119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerDied","Data":"14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4"}
Sep 10 12:33:52.101258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.101142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerDied","Data":"4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33"}
Sep 10 12:33:52.453662 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.453637    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:52.576659 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.576626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-metrics-client-certs\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.576659 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.576657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-metrics-client-ca\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.576871 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.576678    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-0\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.576871 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.576695    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-2\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.576871 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.576715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-serving-certs-ca-bundle\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577022 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.576931    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-grpc-tls\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577078 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-thanos-prometheus-http-client-file\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577078 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:52.577078 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577058    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577228 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:52.577228 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577089    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-web-config\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577228 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-tls-assets\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577228 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-db\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577228 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-kube-rbac-proxy\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-1\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577295    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-trusted-ca-bundle\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577356    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqjll\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-kube-api-access-gqjll\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-kubelet-serving-ca-bundle\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-tls\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config-out\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config\") pod \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\" (UID: \"fbcb3d67-2dc8-4141-982d-f315ccc023a0\") "
Sep 10 12:33:52.577989 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577783    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-metrics-client-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.577989 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577804    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.577989 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.577819    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.578878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.578227    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:52.580220 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.579740    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:52.580220 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.579833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:52.580220 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.579925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.580445 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.580289    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.580445 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.580348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:52.580843 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.580804    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:52.580941 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.580908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.581392 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.581361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.581392 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.581388    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.581511 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.581415    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config" (OuterVolumeSpecName: "config") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.581511 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.581462    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.581948 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.581907    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config-out" (OuterVolumeSpecName: "config-out") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:52.582277 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.582245    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.582827 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.582800    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-kube-api-access-gqjll" (OuterVolumeSpecName: "kube-api-access-gqjll") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "kube-api-access-gqjll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:52.582984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.582964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:52.591070 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.591044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-web-config" (OuterVolumeSpecName: "web-config") pod "fbcb3d67-2dc8-4141-982d-f315ccc023a0" (UID: "fbcb3d67-2dc8-4141-982d-f315ccc023a0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:52.678618 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678545    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678618 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678569    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678618 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678581    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config-out\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678618 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678590    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-config\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678622    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-metrics-client-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678636    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678649    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-grpc-tls\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678659    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678667    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678698    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-web-config\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678708    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-tls-assets\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678716    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-db\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678724    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-kube-rbac-proxy\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678733    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbcb3d67-2dc8-4141-982d-f315ccc023a0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678741    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678749    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbcb3d67-2dc8-4141-982d-f315ccc023a0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:52.678808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:52.678756    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqjll\" (UniqueName: \"kubernetes.io/projected/fbcb3d67-2dc8-4141-982d-f315ccc023a0-kube-api-access-gqjll\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:33:53.107888 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.107856    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerID="f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7" exitCode=0
Sep 10 12:33:53.108045 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.107924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerDied","Data":"f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7"}
Sep 10 12:33:53.108045 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.107936    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.108045 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.107977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fbcb3d67-2dc8-4141-982d-f315ccc023a0","Type":"ContainerDied","Data":"ce6c4a37e51e4772911e0e17c963d26783d08e5ce4c255ef97e944140c9c7e28"}
Sep 10 12:33:53.108045 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.108009    2577 scope.go:117] "RemoveContainer" containerID="0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b"
Sep 10 12:33:53.117104 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.117020    2577 scope.go:117] "RemoveContainer" containerID="83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c"
Sep 10 12:33:53.124659 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.124643    2577 scope.go:117] "RemoveContainer" containerID="f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7"
Sep 10 12:33:53.131541 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.131523    2577 scope.go:117] "RemoveContainer" containerID="f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be"
Sep 10 12:33:53.137766 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.137747    2577 scope.go:117] "RemoveContainer" containerID="14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4"
Sep 10 12:33:53.143991 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.143975    2577 scope.go:117] "RemoveContainer" containerID="4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33"
Sep 10 12:33:53.150645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.150624    2577 scope.go:117] "RemoveContainer" containerID="347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a"
Sep 10 12:33:53.156835 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.156818    2577 scope.go:117] "RemoveContainer" containerID="0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b"
Sep 10 12:33:53.157103 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:53.157085    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b\": container with ID starting with 0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b not found: ID does not exist" containerID="0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b"
Sep 10 12:33:53.157150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.157111    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b"} err="failed to get container status \"0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b\": rpc error: code = NotFound desc = could not find container \"0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b\": container with ID starting with 0c3a227a4c488f66132d30a9d28730a26ab1c977645df60e746e7d36ce53da0b not found: ID does not exist"
Sep 10 12:33:53.157150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.157129    2577 scope.go:117] "RemoveContainer" containerID="83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c"
Sep 10 12:33:53.157350 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:53.157335    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c\": container with ID starting with 83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c not found: ID does not exist" containerID="83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c"
Sep 10 12:33:53.157395 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.157354    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c"} err="failed to get container status \"83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c\": rpc error: code = NotFound desc = could not find container \"83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c\": container with ID starting with 83b0946fd178b60e8aca68fc70a73e4f295dd11965db74c89bd36c577614ce9c not found: ID does not exist"
Sep 10 12:33:53.157395 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.157368    2577 scope.go:117] "RemoveContainer" containerID="f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7"
Sep 10 12:33:53.157576 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:53.157562    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7\": container with ID starting with f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7 not found: ID does not exist" containerID="f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7"
Sep 10 12:33:53.157637 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.157579    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7"} err="failed to get container status \"f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7\": rpc error: code = NotFound desc = could not find container \"f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7\": container with ID starting with f6c994cbe428eed3ba72bd29a39f073569bdf065d98a09f0fdafd1a75b7804e7 not found: ID does not exist"
Sep 10 12:33:53.157637 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.157604    2577 scope.go:117] "RemoveContainer" containerID="f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be"
Sep 10 12:33:53.157857 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:53.157842    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be\": container with ID starting with f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be not found: ID does not exist" containerID="f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be"
Sep 10 12:33:53.157902 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.157860    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be"} err="failed to get container status \"f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be\": rpc error: code = NotFound desc = could not find container \"f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be\": container with ID starting with f7bdc6ce103538bdea0ba7fa1748332ac208f51c0af37c3b40fb03e127e704be not found: ID does not exist"
Sep 10 12:33:53.157902 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.157871    2577 scope.go:117] "RemoveContainer" containerID="14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4"
Sep 10 12:33:53.158075 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:53.158059    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4\": container with ID starting with 14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4 not found: ID does not exist" containerID="14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4"
Sep 10 12:33:53.158111 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.158081    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4"} err="failed to get container status \"14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4\": rpc error: code = NotFound desc = could not find container \"14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4\": container with ID starting with 14e5aa0330ec04239b31db62b4cf05f5ff6f36f3c6a75051afa01cc46acfa6d4 not found: ID does not exist"
Sep 10 12:33:53.158111 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.158097    2577 scope.go:117] "RemoveContainer" containerID="4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33"
Sep 10 12:33:53.158295 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:53.158277    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33\": container with ID starting with 4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33 not found: ID does not exist" containerID="4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33"
Sep 10 12:33:53.158335 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.158298    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33"} err="failed to get container status \"4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33\": rpc error: code = NotFound desc = could not find container \"4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33\": container with ID starting with 4f184e92cec3605ad886fcb3ebd925961f1f195942432c946a82145b675ecd33 not found: ID does not exist"
Sep 10 12:33:53.158335 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.158312    2577 scope.go:117] "RemoveContainer" containerID="347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a"
Sep 10 12:33:53.158505 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:33:53.158489    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a\": container with ID starting with 347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a not found: ID does not exist" containerID="347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a"
Sep 10 12:33:53.158539 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.158508    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a"} err="failed to get container status \"347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a\": rpc error: code = NotFound desc = could not find container \"347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a\": container with ID starting with 347d3e6eaf9a70b7e21926b1c7386efac098ee55461d00e7533826102684a43a not found: ID does not exist"
Sep 10 12:33:53.165098 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.165074    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:33:53.172202 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.172177    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:33:53.233122 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233097    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:33:53.233395 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233380    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="prometheus"
Sep 10 12:33:53.233437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233399    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="prometheus"
Sep 10 12:33:53.233437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233410    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy-web"
Sep 10 12:33:53.233437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233417    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy-web"
Sep 10 12:33:53.233437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233424    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy-thanos"
Sep 10 12:33:53.233437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233430    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy-thanos"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233441    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233447    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233454    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="prom-label-proxy"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233460    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="prom-label-proxy"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233469    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="config-reloader"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233477    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="config-reloader"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233485    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6169f56-1dc5-40a8-b629-7b6356066f8f" containerName="console"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233491    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6169f56-1dc5-40a8-b629-7b6356066f8f" containerName="console"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233496    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="init-config-reloader"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233501    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="init-config-reloader"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233508    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="thanos-sidecar"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233513    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="thanos-sidecar"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233519    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy-metric"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233523    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy-metric"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233530    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="init-config-reloader"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="init-config-reloader"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233542    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233550    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233563    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee5151f-1d8e-4fab-bf05-72582b0287e8" containerName="console"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233569    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee5151f-1d8e-4fab-bf05-72582b0287e8" containerName="console"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233576    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy-web"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233581    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy-web"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233589    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="alertmanager"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233605    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="alertmanager"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233611    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="config-reloader"
Sep 10 12:33:53.233580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233615    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="config-reloader"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233667    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy-web"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233674    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233681    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="prometheus"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233686    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="config-reloader"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233691    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="prom-label-proxy"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233695    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="config-reloader"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233702    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="alertmanager"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233711    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6169f56-1dc5-40a8-b629-7b6356066f8f" containerName="console"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233720    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy-web"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233726    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda45807-387e-443d-bc39-c02b4fcd42b5" containerName="kube-rbac-proxy-metric"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233732    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee5151f-1d8e-4fab-bf05-72582b0287e8" containerName="console"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233737    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="thanos-sidecar"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233741    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy"
Sep 10 12:33:53.234300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.233745    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" containerName="kube-rbac-proxy-thanos"
Sep 10 12:33:53.237854 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.237836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.244822 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.244804    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 12:33:53.245219 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.245204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 12:33:53.246481 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.246463    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 12:33:53.247115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.247094    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 12:33:53.247456 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.247442    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xgnxq\""
Sep 10 12:33:53.247704 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.247688    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 12:33:53.248574 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.248552    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 12:33:53.248697 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.248574    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 12:33:53.248697 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.248574    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 12:33:53.248911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.248894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 12:33:53.248984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.248947    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 12:33:53.248984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.248972    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2g0inaf4ilrs7\""
Sep 10 12:33:53.249488 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.249470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 12:33:53.249550 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.249534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 12:33:53.250498 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.250484    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 12:33:53.255012 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.254997    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 12:33:53.263730 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.263712    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:33:53.284850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.284823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.284948 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.284875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.284948 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.284907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.284948 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.284933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.284976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-config\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.284997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285401 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285401 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285401 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285546 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285546 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9vbp\" (UniqueName: \"kubernetes.io/projected/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-kube-api-access-v9vbp\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285546 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.285546 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.285528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.385916 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.385859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.385916 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.385895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.385916 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.385914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vbp\" (UniqueName: \"kubernetes.io/projected/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-kube-api-access-v9vbp\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.386555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-config\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.387089 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.387089 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.387089 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.386856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.388319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.387271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.388319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.387478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.388319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.387586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.388319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.387975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.389850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.389313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.389850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.389344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.389850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.389429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.389850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.389718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.390261 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.390212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.390451 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.390404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.390875 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.390853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.391224 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.391203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.391781 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.391757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.391995 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.391978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-config\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.392211 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.392188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.392286 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.392274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.392413 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.392396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.397363 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.397342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9vbp\" (UniqueName: \"kubernetes.io/projected/efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8-kube-api-access-v9vbp\") pod \"prometheus-k8s-0\" (UID: \"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.546812 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.546787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:33:53.826450 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.826420    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:33:53.827776 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:33:53.827752    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbd6785_8bec_4ecc_9e6c_cfb008b8d7a8.slice/crio-989bd3b730a2f22708cd1c374bd2556a0102ba3a421fafc10f6f93e02f746ba3 WatchSource:0}: Error finding container 989bd3b730a2f22708cd1c374bd2556a0102ba3a421fafc10f6f93e02f746ba3: Status 404 returned error can't find the container with id 989bd3b730a2f22708cd1c374bd2556a0102ba3a421fafc10f6f93e02f746ba3
Sep 10 12:33:53.878198 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.878165    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:53.998930 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:53.998901    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbcb3d67-2dc8-4141-982d-f315ccc023a0" path="/var/lib/kubelet/pods/fbcb3d67-2dc8-4141-982d-f315ccc023a0/volumes"
Sep 10 12:33:54.113754 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:54.113676    2577 generic.go:356] "Generic (PLEG): container finished" podID="efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8" containerID="a98e5407d278aa48e7046c25a47cba3ef98f68938d060342f95bf27c936fd592" exitCode=0
Sep 10 12:33:54.113754 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:54.113720    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8","Type":"ContainerDied","Data":"a98e5407d278aa48e7046c25a47cba3ef98f68938d060342f95bf27c936fd592"}
Sep 10 12:33:54.113916 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:54.113784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8","Type":"ContainerStarted","Data":"989bd3b730a2f22708cd1c374bd2556a0102ba3a421fafc10f6f93e02f746ba3"}
Sep 10 12:33:54.683072 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:54.682986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:54.788536 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:54.788504    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:33:55.121303 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:55.121271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8","Type":"ContainerStarted","Data":"ab7f7d08b4e609c06a0dd01c0a8e903ced1da4c1d80be6d967d9f6b22e283280"}
Sep 10 12:33:55.121460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:55.121333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8","Type":"ContainerStarted","Data":"545acc5b89c149bac4f24c14a1ad62c468de146f0d8588ac130f238143cc0eea"}
Sep 10 12:33:55.121460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:55.121357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8","Type":"ContainerStarted","Data":"648c14587306c65239ade8824dd79cb4022ecf83a345f1479cebeb607c575e76"}
Sep 10 12:33:55.121460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:55.121381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8","Type":"ContainerStarted","Data":"16d757147d7ac3008ef7b5f22ca92857f7d499c3c777981be96b200dc4f5f4b5"}
Sep 10 12:33:55.121460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:55.121407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8","Type":"ContainerStarted","Data":"9a31fb01ec168ce1a411ac8613705404cc35f2520ea3367ebb7d1b5b2700bc08"}
Sep 10 12:33:55.121460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:55.121430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8","Type":"ContainerStarted","Data":"1c08426a811bb72435c9dc1a9292bc92bac840255f952e1fe50456597df1835b"}
Sep 10 12:33:55.153932 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:55.153888    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.132666911 podStartE2EDuration="2.153871775s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.204864ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:33:55.150964796 +0000 UTC m=+209.673133848" watchObservedRunningTime="2026-09-10 12:33:55.153871775 +0000 UTC m=+209.676040849"
Sep 10 12:33:58.547143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:33:58.547111    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:34:00.898078 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:00.898038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:00.963546 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:00.963519    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:08.170751 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:08.170699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:08.273675 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:08.273627    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:11.538609 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:11.538564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:11.659315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:11.659285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:11.752344 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:11.752308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:51.328414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:51.328312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:51.397426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:51.397382    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:53.547613 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:53.547561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:34:53.562442 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:53.562414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:34:54.368895 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:54.368868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:34:59.180832 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:59.180796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:34:59.280793 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:34:59.280757    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:04.845870 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:04.845823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:04.925296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:04.925255    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:25.143789 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:25.143732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:25.866173 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:25.866126    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:35:25.869291 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:25.869268    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:35:25.871548 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:25.871526    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:35:25.874670 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:25.874644    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:35:25.878224 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:25.878210    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 12:35:26.382667 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:26.382634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:27.201569 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:27.201536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:29.978778 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:29.978737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:30.078828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:30.078793    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:31.767837 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:31.767794    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:31.878658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:31.878620    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:32.055268 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:32.055239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:32.177544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:32.177510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:32.810869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:32.810836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:33.771438 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:33.771407    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:34.159856 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.159773    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:34.262220 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.262193    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"]
Sep 10 12:35:34.264976 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.264959    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"
Sep 10 12:35:34.267471 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.267450    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 12:35:34.268452 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.268434    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 12:35:34.268570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.268440    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4b5bz\""
Sep 10 12:35:34.280081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.280056    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"]
Sep 10 12:35:34.394805 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.394779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8l9s2\" (UID: \"17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"
Sep 10 12:35:34.394958 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.394825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jm9p\" (UniqueName: \"kubernetes.io/projected/17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0-kube-api-access-2jm9p\") pod \"cert-manager-cainjector-559cdd8c87-8l9s2\" (UID: \"17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"
Sep 10 12:35:34.495890 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.495815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8l9s2\" (UID: \"17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"
Sep 10 12:35:34.495890 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.495865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jm9p\" (UniqueName: \"kubernetes.io/projected/17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0-kube-api-access-2jm9p\") pod \"cert-manager-cainjector-559cdd8c87-8l9s2\" (UID: \"17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"
Sep 10 12:35:34.507913 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.507882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8l9s2\" (UID: \"17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"
Sep 10 12:35:34.508023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.507955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jm9p\" (UniqueName: \"kubernetes.io/projected/17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0-kube-api-access-2jm9p\") pod \"cert-manager-cainjector-559cdd8c87-8l9s2\" (UID: \"17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"
Sep 10 12:35:34.589168 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.589142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"
Sep 10 12:35:34.869807 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.869775    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2"]
Sep 10 12:35:34.872986 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:35:34.872960    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17a7fef6_66b5_463a_bd2d_a6ffbe48f6d0.slice/crio-303dc4a9ab3a06591bfbf3e3d4f16d570b6147574e415da5ef099bb4e883984f WatchSource:0}: Error finding container 303dc4a9ab3a06591bfbf3e3d4f16d570b6147574e415da5ef099bb4e883984f: Status 404 returned error can't find the container with id 303dc4a9ab3a06591bfbf3e3d4f16d570b6147574e415da5ef099bb4e883984f
Sep 10 12:35:34.874927 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:34.874910    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:35:35.520656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:35.520610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2" event={"ID":"17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0","Type":"ContainerStarted","Data":"303dc4a9ab3a06591bfbf3e3d4f16d570b6147574e415da5ef099bb4e883984f"}
Sep 10 12:35:36.774162 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:36.774126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:36.876776 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:36.876741    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:38.536224 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:38.536187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2" event={"ID":"17a7fef6-66b5-463a-bd2d-a6ffbe48f6d0","Type":"ContainerStarted","Data":"f2e8e2ef291ecc1575136dfabbe195b40594deed02220a289d0925902a024728"}
Sep 10 12:35:38.561732 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:38.561682    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8l9s2" podStartSLOduration=1.5505618989999999 podStartE2EDuration="4.561663198s" totalImagesPullingTime="3.011101299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:35:38.560990517 +0000 UTC m=+313.083159570" watchObservedRunningTime="2026-09-10 12:35:38.561663198 +0000 UTC m=+313.083832257"
Sep 10 12:35:39.650986 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:39.650942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:39.755579 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:39.755538    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:42.568485 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:42.568442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:49.520280 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:49.520247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:49.647876 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:49.647841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:56.681955 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:56.681915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:35:56.787626 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:35:56.787571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:04.914113 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:04.914075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:04.991524 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:04.991489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:06.707039 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:06.707003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:11.029411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.029379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:11.095821 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.095790    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"]
Sep 10 12:36:11.101269 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.101247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.103905 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.103879    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 12:36:11.104930 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.104905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 12:36:11.105031 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.104910    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:36:11.105031 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.104972    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 12:36:11.105158 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.105124    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ht8q8\""
Sep 10 12:36:11.105158 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.105135    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 12:36:11.134086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.134064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"]
Sep 10 12:36:11.153483 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.153453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:11.201978 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.201952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45c79074-852a-4c3e-9193-a9b359371584-manager-config\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.202066 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.201982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c79074-852a-4c3e-9193-a9b359371584-metrics-cert\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.202066 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.202022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c79074-852a-4c3e-9193-a9b359371584-cert\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.202140 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.202104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p46b\" (UniqueName: \"kubernetes.io/projected/45c79074-852a-4c3e-9193-a9b359371584-kube-api-access-2p46b\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.303467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.303378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c79074-852a-4c3e-9193-a9b359371584-cert\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.303467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.303435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p46b\" (UniqueName: \"kubernetes.io/projected/45c79074-852a-4c3e-9193-a9b359371584-kube-api-access-2p46b\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.303467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.303468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45c79074-852a-4c3e-9193-a9b359371584-manager-config\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.303860 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.303484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c79074-852a-4c3e-9193-a9b359371584-metrics-cert\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.304373 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.304351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45c79074-852a-4c3e-9193-a9b359371584-manager-config\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.306531 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.306483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c79074-852a-4c3e-9193-a9b359371584-cert\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.306681 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.306659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c79074-852a-4c3e-9193-a9b359371584-metrics-cert\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.314391 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.314366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p46b\" (UniqueName: \"kubernetes.io/projected/45c79074-852a-4c3e-9193-a9b359371584-kube-api-access-2p46b\") pod \"lws-controller-manager-5d95565f4b-jxmgb\" (UID: \"45c79074-852a-4c3e-9193-a9b359371584\") " pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.410277 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.410242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:11.692622 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:11.692575    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"]
Sep 10 12:36:11.695081 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:36:11.695052    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c79074_852a_4c3e_9193_a9b359371584.slice/crio-cee8ec19d8fd14dd1297a2b6c459cd5421d86dd54eccdc29434668a046df69bf WatchSource:0}: Error finding container cee8ec19d8fd14dd1297a2b6c459cd5421d86dd54eccdc29434668a046df69bf: Status 404 returned error can't find the container with id cee8ec19d8fd14dd1297a2b6c459cd5421d86dd54eccdc29434668a046df69bf
Sep 10 12:36:12.665299 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:12.665255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb" event={"ID":"45c79074-852a-4c3e-9193-a9b359371584","Type":"ContainerStarted","Data":"cee8ec19d8fd14dd1297a2b6c459cd5421d86dd54eccdc29434668a046df69bf"}
Sep 10 12:36:14.675496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:14.675466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb" event={"ID":"45c79074-852a-4c3e-9193-a9b359371584","Type":"ContainerStarted","Data":"7e058d88423d07500e6b69d57b8083192996ec6515bd4c8dc338ea72bf7c1123"}
Sep 10 12:36:14.675845 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:14.675605    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:14.695204 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:14.695160    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb" podStartSLOduration=1.238849731 podStartE2EDuration="3.695147005s" totalImagesPullingTime="2.456297274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:14.693711901 +0000 UTC m=+349.215880953" watchObservedRunningTime="2026-09-10 12:36:14.695147005 +0000 UTC m=+349.217316056"
Sep 10 12:36:21.743859 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:21.743791    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:21.872035 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:21.872001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:22.226120 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.226085    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:22.703440 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.703403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:22.803118 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.803075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:22.886150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.886116    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"]
Sep 10 12:36:22.889575 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.889553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:22.895200 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.895183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 12:36:22.895200 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.895193    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 12:36:22.895467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.895445    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 12:36:22.895608 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.895538    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-j45z8\""
Sep 10 12:36:22.895608 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.895572    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 12:36:22.952235 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.952207    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"]
Sep 10 12:36:22.997945 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.997881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbj6\" (UniqueName: \"kubernetes.io/projected/e2717e90-2370-4dfe-9da9-7878f53e4df9-kube-api-access-jmbj6\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:22.997945 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.997914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2717e90-2370-4dfe-9da9-7878f53e4df9-manifests\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:22.997945 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.997933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2717e90-2370-4dfe-9da9-7878f53e4df9-tmp\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:22.998119 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.998003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2717e90-2370-4dfe-9da9-7878f53e4df9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:22.998119 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:22.998048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2717e90-2370-4dfe-9da9-7878f53e4df9-webhook-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.098418 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.098389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbj6\" (UniqueName: \"kubernetes.io/projected/e2717e90-2370-4dfe-9da9-7878f53e4df9-kube-api-access-jmbj6\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.098551 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.098436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2717e90-2370-4dfe-9da9-7878f53e4df9-manifests\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.098551 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.098459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2717e90-2370-4dfe-9da9-7878f53e4df9-tmp\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.098551 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.098507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2717e90-2370-4dfe-9da9-7878f53e4df9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.098732 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.098556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2717e90-2370-4dfe-9da9-7878f53e4df9-webhook-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.098893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.098871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2717e90-2370-4dfe-9da9-7878f53e4df9-manifests\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.100952 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.100931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2717e90-2370-4dfe-9da9-7878f53e4df9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.101037 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.100938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2717e90-2370-4dfe-9da9-7878f53e4df9-tmp\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.101292 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.101270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2717e90-2370-4dfe-9da9-7878f53e4df9-webhook-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.110109 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.110073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbj6\" (UniqueName: \"kubernetes.io/projected/e2717e90-2370-4dfe-9da9-7878f53e4df9-kube-api-access-jmbj6\") pod \"opendatahub-operator-controller-manager-669b9dff44-fdj9c\" (UID: \"e2717e90-2370-4dfe-9da9-7878f53e4df9\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.199517 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.199449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:23.199750 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.199729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:23.308181 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.308145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:23.408835 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.408806    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:23.506395 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.506360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"]
Sep 10 12:36:23.509873 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:36:23.509844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2717e90_2370_4dfe_9da9_7878f53e4df9.slice/crio-25130d9aa8ea1dc170366b371f3d09da6220afe48e9ec6313501b096c91dc6d1 WatchSource:0}: Error finding container 25130d9aa8ea1dc170366b371f3d09da6220afe48e9ec6313501b096c91dc6d1: Status 404 returned error can't find the container with id 25130d9aa8ea1dc170366b371f3d09da6220afe48e9ec6313501b096c91dc6d1
Sep 10 12:36:23.547009 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.546939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:23.678370 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.678341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:23.714431 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.714404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c" event={"ID":"e2717e90-2370-4dfe-9da9-7878f53e4df9","Type":"ContainerStarted","Data":"25130d9aa8ea1dc170366b371f3d09da6220afe48e9ec6313501b096c91dc6d1"}
Sep 10 12:36:23.775026 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:23.774998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:25.685097 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:25.685064    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d95565f4b-jxmgb"
Sep 10 12:36:26.731983 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:26.731899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c" event={"ID":"e2717e90-2370-4dfe-9da9-7878f53e4df9","Type":"ContainerStarted","Data":"bf27ebd919dbc4b97f9a193e63f08feb17892b87b7e0fa4e6dba45a755843ecf"}
Sep 10 12:36:27.738256 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:27.738220    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2717e90-2370-4dfe-9da9-7878f53e4df9" containerID="bf27ebd919dbc4b97f9a193e63f08feb17892b87b7e0fa4e6dba45a755843ecf" exitCode=0
Sep 10 12:36:27.738663 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:27.738307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c" event={"ID":"e2717e90-2370-4dfe-9da9-7878f53e4df9","Type":"ContainerDied","Data":"bf27ebd919dbc4b97f9a193e63f08feb17892b87b7e0fa4e6dba45a755843ecf"}
Sep 10 12:36:28.750041 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:28.750007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c" event={"ID":"e2717e90-2370-4dfe-9da9-7878f53e4df9","Type":"ContainerStarted","Data":"36f0c248c19d055b7f544bcad65b93b6dd8a0262909a91a9afa1f4ddea2e787d"}
Sep 10 12:36:28.750467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:28.750221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:28.779850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:28.779661    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c" podStartSLOduration=2.351844108 podStartE2EDuration="6.779642492s" totalImagesPullingTime="3.354458311s" totalInitContainerRuntime="1.073340073s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:36:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:28.778176059 +0000 UTC m=+363.300345114" watchObservedRunningTime="2026-09-10 12:36:28.779642492 +0000 UTC m=+363.301811556"
Sep 10 12:36:39.757142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:39.757113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-fdj9c"
Sep 10 12:36:41.878556 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:41.878521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:41.995832 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:41.995800    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:42.295748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.295718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:42.487766 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.487733    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"]
Sep 10 12:36:42.497987 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.497956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.500551 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.500530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c2qxn\""
Sep 10 12:36:42.501133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.501115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 12:36:42.501636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.501619    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 12:36:42.513041 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.513018    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"]
Sep 10 12:36:42.554139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.554062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4dd94ead-5235-45ac-a072-49282503eb00-tmp\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.554245 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.554177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86ht\" (UniqueName: \"kubernetes.io/projected/4dd94ead-5235-45ac-a072-49282503eb00-kube-api-access-m86ht\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.554245 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.554211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dd94ead-5235-45ac-a072-49282503eb00-tls-certs\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.593496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.593471    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:42.655689 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.655653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86ht\" (UniqueName: \"kubernetes.io/projected/4dd94ead-5235-45ac-a072-49282503eb00-kube-api-access-m86ht\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.655689 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.655697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dd94ead-5235-45ac-a072-49282503eb00-tls-certs\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.655885 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.655735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4dd94ead-5235-45ac-a072-49282503eb00-tmp\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.658083 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.658055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4dd94ead-5235-45ac-a072-49282503eb00-tmp\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.658283 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.658266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dd94ead-5235-45ac-a072-49282503eb00-tls-certs\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.665310 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.665283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86ht\" (UniqueName: \"kubernetes.io/projected/4dd94ead-5235-45ac-a072-49282503eb00-kube-api-access-m86ht\") pod \"kube-auth-proxy-669df89dc4-lrvxv\" (UID: \"4dd94ead-5235-45ac-a072-49282503eb00\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:42.808880 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:42.808790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"
Sep 10 12:36:43.083167 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:43.083090    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv"]
Sep 10 12:36:43.086070 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:36:43.086036    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd94ead_5235_45ac_a072_49282503eb00.slice/crio-cc99dcb12e64f09ac45fb3d12fa77c00d987b539234fc8bd7a010d880bad058f WatchSource:0}: Error finding container cc99dcb12e64f09ac45fb3d12fa77c00d987b539234fc8bd7a010d880bad058f: Status 404 returned error can't find the container with id cc99dcb12e64f09ac45fb3d12fa77c00d987b539234fc8bd7a010d880bad058f
Sep 10 12:36:43.818293 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:43.818251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv" event={"ID":"4dd94ead-5235-45ac-a072-49282503eb00","Type":"ContainerStarted","Data":"cc99dcb12e64f09ac45fb3d12fa77c00d987b539234fc8bd7a010d880bad058f"}
Sep 10 12:36:44.174444 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:44.174364    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:44.275938 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:44.275831    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:45.763575 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:45.763531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:46.834746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:46.834711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv" event={"ID":"4dd94ead-5235-45ac-a072-49282503eb00","Type":"ContainerStarted","Data":"e0a7256211c4e78d14021cab85ff988d8628bdc741c87d2edab458909a55b8ea"}
Sep 10 12:36:46.853621 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:46.853558    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-669df89dc4-lrvxv" podStartSLOduration=1.96723797 podStartE2EDuration="4.853542742s" totalImagesPullingTime="2.886304772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:46.851561229 +0000 UTC m=+381.373730299" watchObservedRunningTime="2026-09-10 12:36:46.853542742 +0000 UTC m=+381.375711839"
Sep 10 12:36:49.961201 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:49.961168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:50.063691 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:50.063661    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:50.768465 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:50.768434    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:50.873620 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:50.873567    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:50.974148 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:50.974117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:52.794208 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:52.794165    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:52.894961 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:52.894920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:53.095748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:53.095658    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:53.311042 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:53.311005    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:36:53.566019 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:36:53.565991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:03.086304 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:03.086249    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:03.196092 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:03.196063    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:03.395517 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:03.395439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:03.508115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:03.508079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:04.425539 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:04.425504    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:04.566512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:04.566484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:04.652271 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:04.652243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:15.895283 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:15.895251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:16.001719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:16.001692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:18.697755 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.697717    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c666f94d8-d54f4"]
Sep 10 12:37:18.703922 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.703897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.707984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.707959    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 12:37:18.708422 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.708403    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 12:37:18.710142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.710118    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7wrxb\""
Sep 10 12:37:18.710142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.710138    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 12:37:18.710289 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.710189    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 12:37:18.712707 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.712663    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 12:37:18.715084 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.715064    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 12:37:18.729803 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.729776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c666f94d8-d54f4"]
Sep 10 12:37:18.823768 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.823740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:18.862719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.862686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-serving-cert\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.862880 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.862733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-service-ca\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.862880 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.862845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-oauth-config\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.862880 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.862868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-config\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.863003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.862894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-trusted-ca-bundle\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.863003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.862953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-oauth-serving-cert\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.863003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.862980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mmt\" (UniqueName: \"kubernetes.io/projected/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-kube-api-access-g5mmt\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.945293 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.945266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:18.964049 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.963995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-serving-cert\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.964049 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-service-ca\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.964247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-oauth-config\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.964247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-config\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.964247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-trusted-ca-bundle\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.964247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-oauth-serving-cert\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.964247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mmt\" (UniqueName: \"kubernetes.io/projected/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-kube-api-access-g5mmt\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.964933 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-oauth-serving-cert\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.965048 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-config\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.965048 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.964974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-service-ca\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.965221 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.965199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-trusted-ca-bundle\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.966704 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.966680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-oauth-config\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:18.966791 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:18.966684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-console-serving-cert\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:19.001097 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:19.001071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mmt\" (UniqueName: \"kubernetes.io/projected/93c3c99d-52c4-4ae7-94a3-c5f7252acd7e-kube-api-access-g5mmt\") pod \"console-6c666f94d8-d54f4\" (UID: \"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e\") " pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:19.014300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:19.014283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:19.346829 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:19.346798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c666f94d8-d54f4"]
Sep 10 12:37:19.353365 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:37:19.353333    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c3c99d_52c4_4ae7_94a3_c5f7252acd7e.slice/crio-815cb07dd12aacb6c6846b2f5e2f1fd7072832514c8531a31a1029f1762e6597 WatchSource:0}: Error finding container 815cb07dd12aacb6c6846b2f5e2f1fd7072832514c8531a31a1029f1762e6597: Status 404 returned error can't find the container with id 815cb07dd12aacb6c6846b2f5e2f1fd7072832514c8531a31a1029f1762e6597
Sep 10 12:37:19.983970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:19.983923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c666f94d8-d54f4" event={"ID":"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e","Type":"ContainerStarted","Data":"da466534a9019dddf0f01e7b809101d9b3aa4877ba17f9914402be5979fa6e64"}
Sep 10 12:37:19.984358 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:19.983984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c666f94d8-d54f4" event={"ID":"93c3c99d-52c4-4ae7-94a3-c5f7252acd7e","Type":"ContainerStarted","Data":"815cb07dd12aacb6c6846b2f5e2f1fd7072832514c8531a31a1029f1762e6597"}
Sep 10 12:37:20.025813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:20.025764    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c666f94d8-d54f4" podStartSLOduration=2.025747246 podStartE2EDuration="2.025747246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:37:20.024219397 +0000 UTC m=+414.546388459" watchObservedRunningTime="2026-09-10 12:37:20.025747246 +0000 UTC m=+414.547916385"
Sep 10 12:37:29.015181 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:29.015137    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:29.015714 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:29.015216    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:29.019898 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:29.019874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:29.028279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:29.028257    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c666f94d8-d54f4"
Sep 10 12:37:29.239493 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:29.239461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:29.339754 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:29.339725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:29.452612 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:29.452576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:54.021459 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:54.021326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:37:54.120405 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:37:54.120372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:10.292245 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:10.292195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:11.290704 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:11.290672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:15.513104 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:15.513073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:16.084824 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:16.084795    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:16.699014 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:16.698976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:21.707196 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:21.707157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:24.694368 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:24.694338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:26.934521 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:26.934488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:27.037159 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:27.037126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:27.724784 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:27.724746    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:27.884423 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:27.884386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:27.988044 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:27.987952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:28.148735 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:28.148705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:28.256215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:28.256139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:28.350174 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:28.350126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:28.453868 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:28.453837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:29.320742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:29.320709    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:30.392118 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:30.392082    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:30.487220 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:30.487193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:33.505920 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:33.505891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:33.605233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:33.605184    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:33.948800 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:33.948765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:34.048817 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:34.048781    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:34.295190 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:34.295145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:34.397388 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:34.397359    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:40.086621 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:40.086549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:40.247398 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:40.247368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:44.721992 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:44.721913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:44.830589 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:44.830556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:47.863402 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:47.863364    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:47.965683 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:47.965644    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:52.646044 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:52.646009    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:52.736064 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:52.736035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:52.838412 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:52.838380    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:59.104968 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:59.104935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:38:59.203082 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:38:59.203054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:03.874349 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:03.874311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:03.974339 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:03.974305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:07.590552 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:07.590515    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:07.692131 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:07.692099    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:07.763797 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:07.763736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:08.263838 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:08.263808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:08.364220 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:08.364188    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:09.323679 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:09.323651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:09.731010 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:09.730934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:12.701544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:12.701514    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:12.794078 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:12.794044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:13.119513 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:13.119483    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:13.221918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:13.221886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:13.457772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:13.457693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:13.557958 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:13.557930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:14.145036 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.145001    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tp9wz"]
Sep 10 12:39:14.148737 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.148721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:14.151678 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.151632    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 12:39:14.151823 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.151800    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 12:39:14.151901 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.151824    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 12:39:14.152646 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.152625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nwphl\""
Sep 10 12:39:14.157218 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.157193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tp9wz"]
Sep 10 12:39:14.219745 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.219716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:14.240654 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.240621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tp9wz"]
Sep 10 12:39:14.268672 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.268639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-config-file\") pod \"limitador-limitador-8588749977-tp9wz\" (UID: \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\") " pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:14.268822 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.268678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxf57\" (UniqueName: \"kubernetes.io/projected/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-kube-api-access-vxf57\") pod \"limitador-limitador-8588749977-tp9wz\" (UID: \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\") " pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:14.304193 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.304161    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:14.369187 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.369159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxf57\" (UniqueName: \"kubernetes.io/projected/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-kube-api-access-vxf57\") pod \"limitador-limitador-8588749977-tp9wz\" (UID: \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\") " pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:14.369333 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.369281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-config-file\") pod \"limitador-limitador-8588749977-tp9wz\" (UID: \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\") " pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:14.369904 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.369883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-config-file\") pod \"limitador-limitador-8588749977-tp9wz\" (UID: \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\") " pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:14.379534 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.379509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxf57\" (UniqueName: \"kubernetes.io/projected/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-kube-api-access-vxf57\") pod \"limitador-limitador-8588749977-tp9wz\" (UID: \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\") " pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:14.459074 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.459020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:14.730965 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:14.730890    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tp9wz"]
Sep 10 12:39:14.734117 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:39:14.734092    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec51bf6_ad50_49d7_aeb2_9e61e51130c6.slice/crio-6b6b26f15775dad5b83077f9b8cdf1e9dc4b739b386e2a2959d2a1e4795e04e4 WatchSource:0}: Error finding container 6b6b26f15775dad5b83077f9b8cdf1e9dc4b739b386e2a2959d2a1e4795e04e4: Status 404 returned error can't find the container with id 6b6b26f15775dad5b83077f9b8cdf1e9dc4b739b386e2a2959d2a1e4795e04e4
Sep 10 12:39:15.061932 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.061902    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8vz65"]
Sep 10 12:39:15.066497 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.066477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65"
Sep 10 12:39:15.068910 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.068889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mjgwg\""
Sep 10 12:39:15.074284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.074226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8vz65"]
Sep 10 12:39:15.105661 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.105637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:15.176507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.176478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psq2w\" (UniqueName: \"kubernetes.io/projected/1a52f576-7fe4-4255-8fad-abbd4e411be6-kube-api-access-psq2w\") pod \"authorino-7bcfd8f9cd-8vz65\" (UID: \"1a52f576-7fe4-4255-8fad-abbd4e411be6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65"
Sep 10 12:39:15.201503 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.201476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:15.202299 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.202277    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-slb8r"]
Sep 10 12:39:15.205747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.205730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-slb8r"
Sep 10 12:39:15.213267 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.213240    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-slb8r"]
Sep 10 12:39:15.277444 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.277415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcx7\" (UniqueName: \"kubernetes.io/projected/1b1ca00d-54c4-44bf-8098-3d0de428e807-kube-api-access-jkcx7\") pod \"authorino-59c974bd48-slb8r\" (UID: \"1b1ca00d-54c4-44bf-8098-3d0de428e807\") " pod="kuadrant-system/authorino-59c974bd48-slb8r"
Sep 10 12:39:15.277636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.277465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psq2w\" (UniqueName: \"kubernetes.io/projected/1a52f576-7fe4-4255-8fad-abbd4e411be6-kube-api-access-psq2w\") pod \"authorino-7bcfd8f9cd-8vz65\" (UID: \"1a52f576-7fe4-4255-8fad-abbd4e411be6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65"
Sep 10 12:39:15.286094 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.286068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psq2w\" (UniqueName: \"kubernetes.io/projected/1a52f576-7fe4-4255-8fad-abbd4e411be6-kube-api-access-psq2w\") pod \"authorino-7bcfd8f9cd-8vz65\" (UID: \"1a52f576-7fe4-4255-8fad-abbd4e411be6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65"
Sep 10 12:39:15.377251 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.377165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65"
Sep 10 12:39:15.378486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.378463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcx7\" (UniqueName: \"kubernetes.io/projected/1b1ca00d-54c4-44bf-8098-3d0de428e807-kube-api-access-jkcx7\") pod \"authorino-59c974bd48-slb8r\" (UID: \"1b1ca00d-54c4-44bf-8098-3d0de428e807\") " pod="kuadrant-system/authorino-59c974bd48-slb8r"
Sep 10 12:39:15.387431 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.387412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcx7\" (UniqueName: \"kubernetes.io/projected/1b1ca00d-54c4-44bf-8098-3d0de428e807-kube-api-access-jkcx7\") pod \"authorino-59c974bd48-slb8r\" (UID: \"1b1ca00d-54c4-44bf-8098-3d0de428e807\") " pod="kuadrant-system/authorino-59c974bd48-slb8r"
Sep 10 12:39:15.509663 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.509627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz" event={"ID":"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6","Type":"ContainerStarted","Data":"6b6b26f15775dad5b83077f9b8cdf1e9dc4b739b386e2a2959d2a1e4795e04e4"}
Sep 10 12:39:15.515740 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.515722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-slb8r"
Sep 10 12:39:15.648627 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.648576    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8vz65"]
Sep 10 12:39:15.652102 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:39:15.651792    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a52f576_7fe4_4255_8fad_abbd4e411be6.slice/crio-a1cd33080a508243701950b6442f667716514a282f017808084cd87f777d30f9 WatchSource:0}: Error finding container a1cd33080a508243701950b6442f667716514a282f017808084cd87f777d30f9: Status 404 returned error can't find the container with id a1cd33080a508243701950b6442f667716514a282f017808084cd87f777d30f9
Sep 10 12:39:15.797705 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:15.797680    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-slb8r"]
Sep 10 12:39:15.800531 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:39:15.800504    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b1ca00d_54c4_44bf_8098_3d0de428e807.slice/crio-8f99d8b0f22c150d83957b67ee4336cba16fbe3a0013d078b011d36b1953f62e WatchSource:0}: Error finding container 8f99d8b0f22c150d83957b67ee4336cba16fbe3a0013d078b011d36b1953f62e: Status 404 returned error can't find the container with id 8f99d8b0f22c150d83957b67ee4336cba16fbe3a0013d078b011d36b1953f62e
Sep 10 12:39:16.516269 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:16.516225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-slb8r" event={"ID":"1b1ca00d-54c4-44bf-8098-3d0de428e807","Type":"ContainerStarted","Data":"8f99d8b0f22c150d83957b67ee4336cba16fbe3a0013d078b011d36b1953f62e"}
Sep 10 12:39:16.517536 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:16.517504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65" event={"ID":"1a52f576-7fe4-4255-8fad-abbd4e411be6","Type":"ContainerStarted","Data":"a1cd33080a508243701950b6442f667716514a282f017808084cd87f777d30f9"}
Sep 10 12:39:17.527281 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:17.526876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz" event={"ID":"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6","Type":"ContainerStarted","Data":"c9e775c9def2b6a72e2573e519010b2f5014ed9f095ff410137d71e70f828f19"}
Sep 10 12:39:17.527281 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:17.527008    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:17.549279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:17.549228    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz" podStartSLOduration=1.185916853 podStartE2EDuration="3.549212818s" totalImagesPullingTime="2.363295965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:39:17.5474486 +0000 UTC m=+532.069617654" watchObservedRunningTime="2026-09-10 12:39:17.549212818 +0000 UTC m=+532.071381870"
Sep 10 12:39:18.535506 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:18.535467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-slb8r" event={"ID":"1b1ca00d-54c4-44bf-8098-3d0de428e807","Type":"ContainerStarted","Data":"f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677"}
Sep 10 12:39:18.556006 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:18.555933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-slb8r" podStartSLOduration=0.967455486 podStartE2EDuration="3.555913129s" totalImagesPullingTime="2.588457643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:39:18.553565544 +0000 UTC m=+533.075734589" watchObservedRunningTime="2026-09-10 12:39:18.555913129 +0000 UTC m=+533.078082182"
Sep 10 12:39:18.583189 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:18.583155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8vz65"]
Sep 10 12:39:18.949300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:18.949269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:19.049588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:19.049555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:19.541995 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:19.541961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65" event={"ID":"1a52f576-7fe4-4255-8fad-abbd4e411be6","Type":"ContainerStarted","Data":"de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8"}
Sep 10 12:39:19.542423 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:19.542050    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65" podUID="1a52f576-7fe4-4255-8fad-abbd4e411be6" containerName="authorino" containerID="cri-o://de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8" gracePeriod=30
Sep 10 12:39:19.561368 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:19.561326    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65" podStartSLOduration=1.4554916279999999 podStartE2EDuration="4.561311476s" totalImagesPullingTime="3.105819848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:39:19.560053754 +0000 UTC m=+534.082222820" watchObservedRunningTime="2026-09-10 12:39:19.561311476 +0000 UTC m=+534.083480527"
Sep 10 12:39:19.778654 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:19.778631    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65"
Sep 10 12:39:19.920544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:19.920466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psq2w\" (UniqueName: \"kubernetes.io/projected/1a52f576-7fe4-4255-8fad-abbd4e411be6-kube-api-access-psq2w\") pod \"1a52f576-7fe4-4255-8fad-abbd4e411be6\" (UID: \"1a52f576-7fe4-4255-8fad-abbd4e411be6\") "
Sep 10 12:39:19.922651 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:19.922622    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a52f576-7fe4-4255-8fad-abbd4e411be6-kube-api-access-psq2w" (OuterVolumeSpecName: "kube-api-access-psq2w") pod "1a52f576-7fe4-4255-8fad-abbd4e411be6" (UID: "1a52f576-7fe4-4255-8fad-abbd4e411be6"). InnerVolumeSpecName "kube-api-access-psq2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:39:20.021296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.021261    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psq2w\" (UniqueName: \"kubernetes.io/projected/1a52f576-7fe4-4255-8fad-abbd4e411be6-kube-api-access-psq2w\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:39:20.553376 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.553340    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a52f576-7fe4-4255-8fad-abbd4e411be6" containerID="de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8" exitCode=0
Sep 10 12:39:20.553858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.553404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65"
Sep 10 12:39:20.553858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.553415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65" event={"ID":"1a52f576-7fe4-4255-8fad-abbd4e411be6","Type":"ContainerDied","Data":"de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8"}
Sep 10 12:39:20.553858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.553457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8vz65" event={"ID":"1a52f576-7fe4-4255-8fad-abbd4e411be6","Type":"ContainerDied","Data":"a1cd33080a508243701950b6442f667716514a282f017808084cd87f777d30f9"}
Sep 10 12:39:20.553858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.553481    2577 scope.go:117] "RemoveContainer" containerID="de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8"
Sep 10 12:39:20.561606 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.561579    2577 scope.go:117] "RemoveContainer" containerID="de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8"
Sep 10 12:39:20.561871 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:39:20.561852    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8\": container with ID starting with de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8 not found: ID does not exist" containerID="de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8"
Sep 10 12:39:20.561921 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.561879    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8"} err="failed to get container status \"de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8\": rpc error: code = NotFound desc = could not find container \"de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8\": container with ID starting with de74bf6e2c27c3742cc15ad62281e555354b3260f7872d2b1671d517b7d9cca8 not found: ID does not exist"
Sep 10 12:39:20.573185 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.573160    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8vz65"]
Sep 10 12:39:20.577983 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:20.577961    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8vz65"]
Sep 10 12:39:22.000540 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:22.000506    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a52f576-7fe4-4255-8fad-abbd4e411be6" path="/var/lib/kubelet/pods/1a52f576-7fe4-4255-8fad-abbd4e411be6/volumes"
Sep 10 12:39:23.714267 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:23.714232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:23.813764 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:23.813736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:28.112756 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.112724    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tp9wz"]
Sep 10 12:39:28.113153 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.112937    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz" podUID="0ec51bf6-ad50-49d7-aeb2-9e61e51130c6" containerName="limitador" containerID="cri-o://c9e775c9def2b6a72e2573e519010b2f5014ed9f095ff410137d71e70f828f19" gracePeriod=30
Sep 10 12:39:28.113661 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.113580    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:28.593758 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.593733    2577 generic.go:356] "Generic (PLEG): container finished" podID="0ec51bf6-ad50-49d7-aeb2-9e61e51130c6" containerID="c9e775c9def2b6a72e2573e519010b2f5014ed9f095ff410137d71e70f828f19" exitCode=0
Sep 10 12:39:28.593889 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.593808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz" event={"ID":"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6","Type":"ContainerDied","Data":"c9e775c9def2b6a72e2573e519010b2f5014ed9f095ff410137d71e70f828f19"}
Sep 10 12:39:28.649539 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.649517    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:28.797482 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.797450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxf57\" (UniqueName: \"kubernetes.io/projected/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-kube-api-access-vxf57\") pod \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\" (UID: \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\") "
Sep 10 12:39:28.797677 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.797637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-config-file\") pod \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\" (UID: \"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6\") "
Sep 10 12:39:28.797940 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.797919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-config-file" (OuterVolumeSpecName: "config-file") pod "0ec51bf6-ad50-49d7-aeb2-9e61e51130c6" (UID: "0ec51bf6-ad50-49d7-aeb2-9e61e51130c6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:39:28.799451 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.799428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-kube-api-access-vxf57" (OuterVolumeSpecName: "kube-api-access-vxf57") pod "0ec51bf6-ad50-49d7-aeb2-9e61e51130c6" (UID: "0ec51bf6-ad50-49d7-aeb2-9e61e51130c6"). InnerVolumeSpecName "kube-api-access-vxf57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:39:28.898640 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.898590    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxf57\" (UniqueName: \"kubernetes.io/projected/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-kube-api-access-vxf57\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:39:28.898640 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:28.898636    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6-config-file\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:39:29.599982 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:29.599952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz"
Sep 10 12:39:29.600386 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:29.599986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tp9wz" event={"ID":"0ec51bf6-ad50-49d7-aeb2-9e61e51130c6","Type":"ContainerDied","Data":"6b6b26f15775dad5b83077f9b8cdf1e9dc4b739b386e2a2959d2a1e4795e04e4"}
Sep 10 12:39:29.600386 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:29.600049    2577 scope.go:117] "RemoveContainer" containerID="c9e775c9def2b6a72e2573e519010b2f5014ed9f095ff410137d71e70f828f19"
Sep 10 12:39:29.622076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:29.622051    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tp9wz"]
Sep 10 12:39:29.624179 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:29.624153    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tp9wz"]
Sep 10 12:39:30.000142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.000056    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec51bf6-ad50-49d7-aeb2-9e61e51130c6" path="/var/lib/kubelet/pods/0ec51bf6-ad50-49d7-aeb2-9e61e51130c6/volumes"
Sep 10 12:39:30.223280 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.223247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:30.360416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.360387    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:30.641880 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.641797    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"]
Sep 10 12:39:30.642282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.642139    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a52f576-7fe4-4255-8fad-abbd4e411be6" containerName="authorino"
Sep 10 12:39:30.642282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.642151    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a52f576-7fe4-4255-8fad-abbd4e411be6" containerName="authorino"
Sep 10 12:39:30.642282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.642160    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec51bf6-ad50-49d7-aeb2-9e61e51130c6" containerName="limitador"
Sep 10 12:39:30.642282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.642166    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec51bf6-ad50-49d7-aeb2-9e61e51130c6" containerName="limitador"
Sep 10 12:39:30.642739 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.642706    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec51bf6-ad50-49d7-aeb2-9e61e51130c6" containerName="limitador"
Sep 10 12:39:30.642867 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.642743    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a52f576-7fe4-4255-8fad-abbd4e411be6" containerName="authorino"
Sep 10 12:39:30.651942 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.651894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:30.654657 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.654627    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 12:39:30.654773 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.654664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:39:30.654773 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.654664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 12:39:30.655652 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.655633    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-49wbp\""
Sep 10 12:39:30.657893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.657869    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"]
Sep 10 12:39:30.691128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.691105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:30.814153 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.814122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07b56097-bd8f-4f0d-abc8-93833838c026-data\") pod \"postgres-69fb4594cf-qqc9h\" (UID: \"07b56097-bd8f-4f0d-abc8-93833838c026\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:30.814307 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.814203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7hm\" (UniqueName: \"kubernetes.io/projected/07b56097-bd8f-4f0d-abc8-93833838c026-kube-api-access-kq7hm\") pod \"postgres-69fb4594cf-qqc9h\" (UID: \"07b56097-bd8f-4f0d-abc8-93833838c026\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:30.915555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.915474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07b56097-bd8f-4f0d-abc8-93833838c026-data\") pod \"postgres-69fb4594cf-qqc9h\" (UID: \"07b56097-bd8f-4f0d-abc8-93833838c026\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:30.915555 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.915543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7hm\" (UniqueName: \"kubernetes.io/projected/07b56097-bd8f-4f0d-abc8-93833838c026-kube-api-access-kq7hm\") pod \"postgres-69fb4594cf-qqc9h\" (UID: \"07b56097-bd8f-4f0d-abc8-93833838c026\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:30.915846 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.915828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07b56097-bd8f-4f0d-abc8-93833838c026-data\") pod \"postgres-69fb4594cf-qqc9h\" (UID: \"07b56097-bd8f-4f0d-abc8-93833838c026\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:30.932508 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.932480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7hm\" (UniqueName: \"kubernetes.io/projected/07b56097-bd8f-4f0d-abc8-93833838c026-kube-api-access-kq7hm\") pod \"postgres-69fb4594cf-qqc9h\" (UID: \"07b56097-bd8f-4f0d-abc8-93833838c026\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:30.963405 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:30.963384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:31.237926 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:31.237867    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"]
Sep 10 12:39:31.240143 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:39:31.240110    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b56097_bd8f_4f0d_abc8_93833838c026.slice/crio-85404d494da34b2ed13b84e6a6c48eaa25eeabc247196bf30e1d22d8b95c88eb WatchSource:0}: Error finding container 85404d494da34b2ed13b84e6a6c48eaa25eeabc247196bf30e1d22d8b95c88eb: Status 404 returned error can't find the container with id 85404d494da34b2ed13b84e6a6c48eaa25eeabc247196bf30e1d22d8b95c88eb
Sep 10 12:39:31.610992 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:31.610961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h" event={"ID":"07b56097-bd8f-4f0d-abc8-93833838c026","Type":"ContainerStarted","Data":"85404d494da34b2ed13b84e6a6c48eaa25eeabc247196bf30e1d22d8b95c88eb"}
Sep 10 12:39:36.642783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:36.642699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h" event={"ID":"07b56097-bd8f-4f0d-abc8-93833838c026","Type":"ContainerStarted","Data":"52787b5c760a9f0575358b847ceabdf7acdcb7c659f7aa146d8e3ad143370fa4"}
Sep 10 12:39:36.642783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:36.642774    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:36.660006 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:36.659959    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h" podStartSLOduration=1.524066529 podStartE2EDuration="6.659944524s" totalImagesPullingTime="5.135877995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:39:36.658806066 +0000 UTC m=+551.180975117" watchObservedRunningTime="2026-09-10 12:39:36.659944524 +0000 UTC m=+551.182113576"
Sep 10 12:39:42.676639 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:42.676610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qqc9h"
Sep 10 12:39:43.007416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.007337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:43.219891 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.219857    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-4829w"]
Sep 10 12:39:43.226122 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.226105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4829w"
Sep 10 12:39:43.230539 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.230514    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-4829w"]
Sep 10 12:39:43.271519 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.271494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:43.313630 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.313581    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4829w"]
Sep 10 12:39:43.313890 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:39:43.313871    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xqln9], unattached volumes=[], failed to process volumes=[kube-api-access-xqln9]: context canceled" pod="kuadrant-system/authorino-557f876f69-4829w" podUID="9c4410b5-2ab6-45f5-992c-45106ee9bceb"
Sep 10 12:39:43.325209 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.325187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqln9\" (UniqueName: \"kubernetes.io/projected/9c4410b5-2ab6-45f5-992c-45106ee9bceb-kube-api-access-xqln9\") pod \"authorino-557f876f69-4829w\" (UID: \"9c4410b5-2ab6-45f5-992c-45106ee9bceb\") " pod="kuadrant-system/authorino-557f876f69-4829w"
Sep 10 12:39:43.339294 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.339268    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mzppd"]
Sep 10 12:39:43.342789 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.342771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.345284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.345261    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 12:39:43.352975 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.352136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mzppd"]
Sep 10 12:39:43.426560 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.426532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqln9\" (UniqueName: \"kubernetes.io/projected/9c4410b5-2ab6-45f5-992c-45106ee9bceb-kube-api-access-xqln9\") pod \"authorino-557f876f69-4829w\" (UID: \"9c4410b5-2ab6-45f5-992c-45106ee9bceb\") " pod="kuadrant-system/authorino-557f876f69-4829w"
Sep 10 12:39:43.426686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.426633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06a20361-8cef-46c8-a8a1-9cb5f9c99753-tls-cert\") pod \"authorino-6d4dfc649f-mzppd\" (UID: \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\") " pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.426686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.426655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrft5\" (UniqueName: \"kubernetes.io/projected/06a20361-8cef-46c8-a8a1-9cb5f9c99753-kube-api-access-vrft5\") pod \"authorino-6d4dfc649f-mzppd\" (UID: \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\") " pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.434792 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.434760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqln9\" (UniqueName: \"kubernetes.io/projected/9c4410b5-2ab6-45f5-992c-45106ee9bceb-kube-api-access-xqln9\") pod \"authorino-557f876f69-4829w\" (UID: \"9c4410b5-2ab6-45f5-992c-45106ee9bceb\") " pod="kuadrant-system/authorino-557f876f69-4829w"
Sep 10 12:39:43.463430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.463407    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mzppd"]
Sep 10 12:39:43.463630 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:39:43.463611    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vrft5 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-mzppd" podUID="06a20361-8cef-46c8-a8a1-9cb5f9c99753"
Sep 10 12:39:43.494274 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.494249    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79ff54bc7b-h6dcl"]
Sep 10 12:39:43.497901 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.497881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:39:43.509958 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.509936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79ff54bc7b-h6dcl"]
Sep 10 12:39:43.527473 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.527407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06a20361-8cef-46c8-a8a1-9cb5f9c99753-tls-cert\") pod \"authorino-6d4dfc649f-mzppd\" (UID: \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\") " pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.527473 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.527453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrft5\" (UniqueName: \"kubernetes.io/projected/06a20361-8cef-46c8-a8a1-9cb5f9c99753-kube-api-access-vrft5\") pod \"authorino-6d4dfc649f-mzppd\" (UID: \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\") " pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.527636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.527619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67f43d6e-7a6b-4c86-b66a-30305f784362-tls-cert\") pod \"authorino-79ff54bc7b-h6dcl\" (UID: \"67f43d6e-7a6b-4c86-b66a-30305f784362\") " pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:39:43.527684 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.527661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kk2\" (UniqueName: \"kubernetes.io/projected/67f43d6e-7a6b-4c86-b66a-30305f784362-kube-api-access-97kk2\") pod \"authorino-79ff54bc7b-h6dcl\" (UID: \"67f43d6e-7a6b-4c86-b66a-30305f784362\") " pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:39:43.529847 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.529822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06a20361-8cef-46c8-a8a1-9cb5f9c99753-tls-cert\") pod \"authorino-6d4dfc649f-mzppd\" (UID: \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\") " pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.549550 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.549523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrft5\" (UniqueName: \"kubernetes.io/projected/06a20361-8cef-46c8-a8a1-9cb5f9c99753-kube-api-access-vrft5\") pod \"authorino-6d4dfc649f-mzppd\" (UID: \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\") " pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.628488 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.628454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67f43d6e-7a6b-4c86-b66a-30305f784362-tls-cert\") pod \"authorino-79ff54bc7b-h6dcl\" (UID: \"67f43d6e-7a6b-4c86-b66a-30305f784362\") " pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:39:43.628488 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.628489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kk2\" (UniqueName: \"kubernetes.io/projected/67f43d6e-7a6b-4c86-b66a-30305f784362-kube-api-access-97kk2\") pod \"authorino-79ff54bc7b-h6dcl\" (UID: \"67f43d6e-7a6b-4c86-b66a-30305f784362\") " pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:39:43.630686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.630663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67f43d6e-7a6b-4c86-b66a-30305f784362-tls-cert\") pod \"authorino-79ff54bc7b-h6dcl\" (UID: \"67f43d6e-7a6b-4c86-b66a-30305f784362\") " pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:39:43.644523 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.644499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kk2\" (UniqueName: \"kubernetes.io/projected/67f43d6e-7a6b-4c86-b66a-30305f784362-kube-api-access-97kk2\") pod \"authorino-79ff54bc7b-h6dcl\" (UID: \"67f43d6e-7a6b-4c86-b66a-30305f784362\") " pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:39:43.673251 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.673227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.673341 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.673233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4829w"
Sep 10 12:39:43.680397 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.680377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:43.683351 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.683335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4829w"
Sep 10 12:39:43.729704 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.729681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqln9\" (UniqueName: \"kubernetes.io/projected/9c4410b5-2ab6-45f5-992c-45106ee9bceb-kube-api-access-xqln9\") pod \"9c4410b5-2ab6-45f5-992c-45106ee9bceb\" (UID: \"9c4410b5-2ab6-45f5-992c-45106ee9bceb\") "
Sep 10 12:39:43.729814 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.729727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrft5\" (UniqueName: \"kubernetes.io/projected/06a20361-8cef-46c8-a8a1-9cb5f9c99753-kube-api-access-vrft5\") pod \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\" (UID: \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\") "
Sep 10 12:39:43.729814 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.729795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06a20361-8cef-46c8-a8a1-9cb5f9c99753-tls-cert\") pod \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\" (UID: \"06a20361-8cef-46c8-a8a1-9cb5f9c99753\") "
Sep 10 12:39:43.731606 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.731568    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4410b5-2ab6-45f5-992c-45106ee9bceb-kube-api-access-xqln9" (OuterVolumeSpecName: "kube-api-access-xqln9") pod "9c4410b5-2ab6-45f5-992c-45106ee9bceb" (UID: "9c4410b5-2ab6-45f5-992c-45106ee9bceb"). InnerVolumeSpecName "kube-api-access-xqln9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:39:43.731694 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.731667    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a20361-8cef-46c8-a8a1-9cb5f9c99753-kube-api-access-vrft5" (OuterVolumeSpecName: "kube-api-access-vrft5") pod "06a20361-8cef-46c8-a8a1-9cb5f9c99753" (UID: "06a20361-8cef-46c8-a8a1-9cb5f9c99753"). InnerVolumeSpecName "kube-api-access-vrft5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:39:43.731734 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.731693    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a20361-8cef-46c8-a8a1-9cb5f9c99753-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "06a20361-8cef-46c8-a8a1-9cb5f9c99753" (UID: "06a20361-8cef-46c8-a8a1-9cb5f9c99753"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:39:43.807031 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.806979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:39:43.830397 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.830367    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06a20361-8cef-46c8-a8a1-9cb5f9c99753-tls-cert\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:39:43.830397 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.830390    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqln9\" (UniqueName: \"kubernetes.io/projected/9c4410b5-2ab6-45f5-992c-45106ee9bceb-kube-api-access-xqln9\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:39:43.830397 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:43.830400    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrft5\" (UniqueName: \"kubernetes.io/projected/06a20361-8cef-46c8-a8a1-9cb5f9c99753-kube-api-access-vrft5\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:39:44.080247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:44.080220    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79ff54bc7b-h6dcl"]
Sep 10 12:39:44.081761 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:39:44.081734    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67f43d6e_7a6b_4c86_b66a_30305f784362.slice/crio-0aba1af37d54aa573891053e97d5cd72bb8663d10a1da42b43b79f58f87dddf0 WatchSource:0}: Error finding container 0aba1af37d54aa573891053e97d5cd72bb8663d10a1da42b43b79f58f87dddf0: Status 404 returned error can't find the container with id 0aba1af37d54aa573891053e97d5cd72bb8663d10a1da42b43b79f58f87dddf0
Sep 10 12:39:44.679869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:44.679829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl" event={"ID":"67f43d6e-7a6b-4c86-b66a-30305f784362","Type":"ContainerStarted","Data":"0aba1af37d54aa573891053e97d5cd72bb8663d10a1da42b43b79f58f87dddf0"}
Sep 10 12:39:44.679869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:44.679864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mzppd"
Sep 10 12:39:44.680108 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:44.679844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4829w"
Sep 10 12:39:44.711508 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:44.711446    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mzppd"]
Sep 10 12:39:44.715492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:44.715461    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mzppd"]
Sep 10 12:39:44.737874 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:44.737851    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4829w"]
Sep 10 12:39:44.740525 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:44.740502    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-4829w"]
Sep 10 12:39:45.686296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:45.686261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl" event={"ID":"67f43d6e-7a6b-4c86-b66a-30305f784362","Type":"ContainerStarted","Data":"6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8"}
Sep 10 12:39:45.705012 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:45.704967    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl" podStartSLOduration=1.975879408 podStartE2EDuration="2.7049524s" totalImagesPullingTime="729.072992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:39:45.703532186 +0000 UTC m=+560.225701252" watchObservedRunningTime="2026-09-10 12:39:45.7049524 +0000 UTC m=+560.227121482"
Sep 10 12:39:45.732374 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:45.732343    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-slb8r"]
Sep 10 12:39:45.732803 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:45.732545    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-slb8r" podUID="1b1ca00d-54c4-44bf-8098-3d0de428e807" containerName="authorino" containerID="cri-o://f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677" gracePeriod=30
Sep 10 12:39:46.000139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.000110    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06a20361-8cef-46c8-a8a1-9cb5f9c99753" path="/var/lib/kubelet/pods/06a20361-8cef-46c8-a8a1-9cb5f9c99753/volumes"
Sep 10 12:39:46.000359 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.000347    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4410b5-2ab6-45f5-992c-45106ee9bceb" path="/var/lib/kubelet/pods/9c4410b5-2ab6-45f5-992c-45106ee9bceb/volumes"
Sep 10 12:39:46.015750 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.015728    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-slb8r"
Sep 10 12:39:46.047448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.047421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkcx7\" (UniqueName: \"kubernetes.io/projected/1b1ca00d-54c4-44bf-8098-3d0de428e807-kube-api-access-jkcx7\") pod \"1b1ca00d-54c4-44bf-8098-3d0de428e807\" (UID: \"1b1ca00d-54c4-44bf-8098-3d0de428e807\") "
Sep 10 12:39:46.049468 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.049434    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1ca00d-54c4-44bf-8098-3d0de428e807-kube-api-access-jkcx7" (OuterVolumeSpecName: "kube-api-access-jkcx7") pod "1b1ca00d-54c4-44bf-8098-3d0de428e807" (UID: "1b1ca00d-54c4-44bf-8098-3d0de428e807"). InnerVolumeSpecName "kube-api-access-jkcx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:39:46.148822 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.148792    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcx7\" (UniqueName: \"kubernetes.io/projected/1b1ca00d-54c4-44bf-8098-3d0de428e807-kube-api-access-jkcx7\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:39:46.691712 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.691678    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b1ca00d-54c4-44bf-8098-3d0de428e807" containerID="f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677" exitCode=0
Sep 10 12:39:46.691900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.691732    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-slb8r"
Sep 10 12:39:46.691900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.691771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-slb8r" event={"ID":"1b1ca00d-54c4-44bf-8098-3d0de428e807","Type":"ContainerDied","Data":"f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677"}
Sep 10 12:39:46.691900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.691825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-slb8r" event={"ID":"1b1ca00d-54c4-44bf-8098-3d0de428e807","Type":"ContainerDied","Data":"8f99d8b0f22c150d83957b67ee4336cba16fbe3a0013d078b011d36b1953f62e"}
Sep 10 12:39:46.691900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.691849    2577 scope.go:117] "RemoveContainer" containerID="f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677"
Sep 10 12:39:46.700133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.700106    2577 scope.go:117] "RemoveContainer" containerID="f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677"
Sep 10 12:39:46.700449 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:39:46.700430    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677\": container with ID starting with f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677 not found: ID does not exist" containerID="f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677"
Sep 10 12:39:46.700502 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.700457    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677"} err="failed to get container status \"f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677\": rpc error: code = NotFound desc = could not find container \"f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677\": container with ID starting with f43832091f4f5ec1760ff2ae8e0a94140c802630464b1d1381e8baedc38ae677 not found: ID does not exist"
Sep 10 12:39:46.712432 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.712404    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-slb8r"]
Sep 10 12:39:46.713981 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:46.713963    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-slb8r"]
Sep 10 12:39:48.000670 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:48.000640    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1ca00d-54c4-44bf-8098-3d0de428e807" path="/var/lib/kubelet/pods/1b1ca00d-54c4-44bf-8098-3d0de428e807/volumes"
Sep 10 12:39:50.580051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:50.580021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:50.687725 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:50.687695    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:50.813153 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:50.813126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:50.938303 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:50.938228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:54.274307 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:54.274279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:39:54.369096 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:39:54.369060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:25.897228 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:25.897203    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:40:25.897768 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:25.897706    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:40:25.902055 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:25.902037    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:40:25.902515 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:25.902499    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:40:27.196162 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:27.196120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:27.293957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:27.293921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:28.671736 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:28.671705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:28.766607 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:28.766561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:28.871454 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:28.871427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:28.986265 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:28.986175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:29.174125 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.174092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:29.290789 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.290759    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"]
Sep 10 12:40:29.291125 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.291112    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1ca00d-54c4-44bf-8098-3d0de428e807" containerName="authorino"
Sep 10 12:40:29.291170 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.291126    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1ca00d-54c4-44bf-8098-3d0de428e807" containerName="authorino"
Sep 10 12:40:29.291205 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.291180    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1ca00d-54c4-44bf-8098-3d0de428e807" containerName="authorino"
Sep 10 12:40:29.293376 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.293356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:29.296112 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.296063    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fkg52\""
Sep 10 12:40:29.296112 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.296084    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 12:40:29.309054 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.309020    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zc97k"]
Sep 10 12:40:29.311849 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.311823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"]
Sep 10 12:40:29.312005 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.311904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:29.315302 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.315277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:29.322627 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.322580    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zc97k"]
Sep 10 12:40:29.417494 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.417461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb44708b-b4a0-42c0-8fff-592329cef1f5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5cvsq\" (UID: \"fb44708b-b4a0-42c0-8fff-592329cef1f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:29.417688 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.417505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl286\" (UniqueName: \"kubernetes.io/projected/fb44708b-b4a0-42c0-8fff-592329cef1f5-kube-api-access-wl286\") pod \"payload-pre-processing-59fbc47bf8-5cvsq\" (UID: \"fb44708b-b4a0-42c0-8fff-592329cef1f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:29.417688 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.417538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68357e6d-f22c-40cf-b4b4-67bd383505e9-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zc97k\" (UID: \"68357e6d-f22c-40cf-b4b4-67bd383505e9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:29.417688 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.417669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsnpt\" (UniqueName: \"kubernetes.io/projected/68357e6d-f22c-40cf-b4b4-67bd383505e9-kube-api-access-tsnpt\") pod \"payload-processing-5b54b7fbb7-zc97k\" (UID: \"68357e6d-f22c-40cf-b4b4-67bd383505e9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:29.518463 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.518426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsnpt\" (UniqueName: \"kubernetes.io/projected/68357e6d-f22c-40cf-b4b4-67bd383505e9-kube-api-access-tsnpt\") pod \"payload-processing-5b54b7fbb7-zc97k\" (UID: \"68357e6d-f22c-40cf-b4b4-67bd383505e9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:29.518672 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.518478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb44708b-b4a0-42c0-8fff-592329cef1f5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5cvsq\" (UID: \"fb44708b-b4a0-42c0-8fff-592329cef1f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:29.518672 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.518530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl286\" (UniqueName: \"kubernetes.io/projected/fb44708b-b4a0-42c0-8fff-592329cef1f5-kube-api-access-wl286\") pod \"payload-pre-processing-59fbc47bf8-5cvsq\" (UID: \"fb44708b-b4a0-42c0-8fff-592329cef1f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:29.518672 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.518570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68357e6d-f22c-40cf-b4b4-67bd383505e9-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zc97k\" (UID: \"68357e6d-f22c-40cf-b4b4-67bd383505e9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:29.519443 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.519418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb44708b-b4a0-42c0-8fff-592329cef1f5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5cvsq\" (UID: \"fb44708b-b4a0-42c0-8fff-592329cef1f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:29.519489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.519418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68357e6d-f22c-40cf-b4b4-67bd383505e9-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zc97k\" (UID: \"68357e6d-f22c-40cf-b4b4-67bd383505e9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:29.528246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.528223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl286\" (UniqueName: \"kubernetes.io/projected/fb44708b-b4a0-42c0-8fff-592329cef1f5-kube-api-access-wl286\") pod \"payload-pre-processing-59fbc47bf8-5cvsq\" (UID: \"fb44708b-b4a0-42c0-8fff-592329cef1f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:29.528381 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.528316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsnpt\" (UniqueName: \"kubernetes.io/projected/68357e6d-f22c-40cf-b4b4-67bd383505e9-kube-api-access-tsnpt\") pod \"payload-processing-5b54b7fbb7-zc97k\" (UID: \"68357e6d-f22c-40cf-b4b4-67bd383505e9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:29.605137 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.604838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:29.625915 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.625883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:29.888139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.888107    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"]
Sep 10 12:40:29.890900 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:40:29.890872    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb44708b_b4a0_42c0_8fff_592329cef1f5.slice/crio-6ccef2758be26f185339ca461cbdafbfec772ea0418c9a348fc14d168f0b5432 WatchSource:0}: Error finding container 6ccef2758be26f185339ca461cbdafbfec772ea0418c9a348fc14d168f0b5432: Status 404 returned error can't find the container with id 6ccef2758be26f185339ca461cbdafbfec772ea0418c9a348fc14d168f0b5432
Sep 10 12:40:29.916641 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:29.916614    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zc97k"]
Sep 10 12:40:29.919108 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:40:29.919081    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68357e6d_f22c_40cf_b4b4_67bd383505e9.slice/crio-e82b93e494f33c28cdd6a23b0c8520c60a8977b819b77c99c4e71018b769db5b WatchSource:0}: Error finding container e82b93e494f33c28cdd6a23b0c8520c60a8977b819b77c99c4e71018b769db5b: Status 404 returned error can't find the container with id e82b93e494f33c28cdd6a23b0c8520c60a8977b819b77c99c4e71018b769db5b
Sep 10 12:40:30.905383 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:30.905293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k" event={"ID":"68357e6d-f22c-40cf-b4b4-67bd383505e9","Type":"ContainerStarted","Data":"e82b93e494f33c28cdd6a23b0c8520c60a8977b819b77c99c4e71018b769db5b"}
Sep 10 12:40:30.907794 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:30.907713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq" event={"ID":"fb44708b-b4a0-42c0-8fff-592329cef1f5","Type":"ContainerStarted","Data":"6ccef2758be26f185339ca461cbdafbfec772ea0418c9a348fc14d168f0b5432"}
Sep 10 12:40:32.919138 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:32.919100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k" event={"ID":"68357e6d-f22c-40cf-b4b4-67bd383505e9","Type":"ContainerStarted","Data":"99d314f413a06d92b3a08cfb104088e3d65eb67374d3b394d091b822c973a96e"}
Sep 10 12:40:32.919622 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:32.919248    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:32.920701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:32.920678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq" event={"ID":"fb44708b-b4a0-42c0-8fff-592329cef1f5","Type":"ContainerStarted","Data":"d786c5092587748ca3994fb7c11f973fa1a4e9ba8c86fda361a21138c8da8993"}
Sep 10 12:40:32.920819 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:32.920799    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:32.940995 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:32.940942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k" podStartSLOduration=1.911654327 podStartE2EDuration="3.940929331s" totalImagesPullingTime="2.029275004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:40:32.93997036 +0000 UTC m=+607.462139425" watchObservedRunningTime="2026-09-10 12:40:32.940929331 +0000 UTC m=+607.463098381"
Sep 10 12:40:32.961107 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:32.961053    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq" podStartSLOduration=1.89537679 podStartE2EDuration="3.961041063s" totalImagesPullingTime="2.065664273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:40:32.960098627 +0000 UTC m=+607.482267680" watchObservedRunningTime="2026-09-10 12:40:32.961041063 +0000 UTC m=+607.483210114"
Sep 10 12:40:38.903828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:38.903792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:38.975827 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:38.975792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:39.064949 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:39.064895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:39.176508 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:39.176414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:42.988465 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:42.988434    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:43.089967 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:43.089925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:43.932782 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:43.932751    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5cvsq"
Sep 10 12:40:43.932982 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:43.932877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zc97k"
Sep 10 12:40:45.562198 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:45.562154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:40:45.662540 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:40:45.662500    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:41:05.320482 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:05.320442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:41:10.825149 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.825108    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"]
Sep 10 12:41:10.828206 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.828184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:10.830879 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.830856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j76fb\""
Sep 10 12:41:10.831169 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.831153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 12:41:10.831827 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.831808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 12:41:10.831938 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.831810    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 12:41:10.840495 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.840475    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"]
Sep 10 12:41:10.869576 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.869551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:41:10.980108 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.980073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:10.980108 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.980109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8086638-f32f-4c2e-96a5-6394432f98b4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:10.980312 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.980138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:10.980312 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.980237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:10.980312 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.980295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:10.980432 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:10.980346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzsqb\" (UniqueName: \"kubernetes.io/projected/c8086638-f32f-4c2e-96a5-6394432f98b4-kube-api-access-qzsqb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.080987 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.080918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzsqb\" (UniqueName: \"kubernetes.io/projected/c8086638-f32f-4c2e-96a5-6394432f98b4-kube-api-access-qzsqb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.080987 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.080974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.081185 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.080992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8086638-f32f-4c2e-96a5-6394432f98b4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.081185 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.081010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.081185 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.081042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.081185 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.081089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.081400 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.081380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.081449 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.081406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.081484 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.081441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.083215 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.083189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8086638-f32f-4c2e-96a5-6394432f98b4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.083437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.083421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8086638-f32f-4c2e-96a5-6394432f98b4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.089566 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.089545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzsqb\" (UniqueName: \"kubernetes.io/projected/c8086638-f32f-4c2e-96a5-6394432f98b4-kube-api-access-qzsqb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpxpk\" (UID: \"c8086638-f32f-4c2e-96a5-6394432f98b4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.140582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.140558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:11.414495 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.414433    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"]
Sep 10 12:41:11.417023 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:41:11.416991    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8086638_f32f_4c2e_96a5_6394432f98b4.slice/crio-60bc2046c7b3d861ed6c9931132304cf3ad55ef71b5623a7f43a9352a5e2e363 WatchSource:0}: Error finding container 60bc2046c7b3d861ed6c9931132304cf3ad55ef71b5623a7f43a9352a5e2e363: Status 404 returned error can't find the container with id 60bc2046c7b3d861ed6c9931132304cf3ad55ef71b5623a7f43a9352a5e2e363
Sep 10 12:41:11.418759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:11.418739    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:41:12.131753 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:12.131718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk" event={"ID":"c8086638-f32f-4c2e-96a5-6394432f98b4","Type":"ContainerStarted","Data":"60bc2046c7b3d861ed6c9931132304cf3ad55ef71b5623a7f43a9352a5e2e363"}
Sep 10 12:41:13.137586 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:13.137541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk" event={"ID":"c8086638-f32f-4c2e-96a5-6394432f98b4","Type":"ContainerStarted","Data":"9718facd117f8e7f7556ea6366e61ff8121e8c7dee98023d4672ff4c210301ac"}
Sep 10 12:41:13.158853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:13.158805    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk" podStartSLOduration=2.090787924 podStartE2EDuration="3.158789593s" totalImagesPullingTime="1.068001669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:41:13.156923613 +0000 UTC m=+647.679092665" watchObservedRunningTime="2026-09-10 12:41:13.158789593 +0000 UTC m=+647.680958644"
Sep 10 12:41:16.733589 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:16.733547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:41:21.141170 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:21.141134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:21.141585 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:21.141224    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:21.153412 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:21.153388    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:21.183636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:21.183591    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpxpk"
Sep 10 12:41:22.710924 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:22.710890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:41:26.887651 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:26.887619    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:41:44.622878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.622840    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"]
Sep 10 12:41:44.626580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.626560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.629358 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.629336    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 12:41:44.637489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.637468    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"]
Sep 10 12:41:44.668224 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.668196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:41:44.675379 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.675359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.675468 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.675406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826a241c-f0cf-4722-84be-292191eca9c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.675510 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.675482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.675545 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.675520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrx4n\" (UniqueName: \"kubernetes.io/projected/826a241c-f0cf-4722-84be-292191eca9c2-kube-api-access-mrx4n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.675614 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.675586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.675655 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.675630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.776348 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826a241c-f0cf-4722-84be-292191eca9c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.776448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.776448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrx4n\" (UniqueName: \"kubernetes.io/projected/826a241c-f0cf-4722-84be-292191eca9c2-kube-api-access-mrx4n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.776536 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.776536 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.776536 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.776915 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.777033 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.777033 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.776934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.778772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.778748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826a241c-f0cf-4722-84be-292191eca9c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.778887 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.778872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826a241c-f0cf-4722-84be-292191eca9c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.784911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.784887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrx4n\" (UniqueName: \"kubernetes.io/projected/826a241c-f0cf-4722-84be-292191eca9c2-kube-api-access-mrx4n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9\" (UID: \"826a241c-f0cf-4722-84be-292191eca9c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:44.937643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:44.937548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:45.208731 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:45.208706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"]
Sep 10 12:41:45.210851 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:41:45.210824    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826a241c_f0cf_4722_84be_292191eca9c2.slice/crio-3216c9bf3c2d93ee00bdd837769776554cdb41b79f62affcfd78bf657941752a WatchSource:0}: Error finding container 3216c9bf3c2d93ee00bdd837769776554cdb41b79f62affcfd78bf657941752a: Status 404 returned error can't find the container with id 3216c9bf3c2d93ee00bdd837769776554cdb41b79f62affcfd78bf657941752a
Sep 10 12:41:45.285133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:45.285092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9" event={"ID":"826a241c-f0cf-4722-84be-292191eca9c2","Type":"ContainerStarted","Data":"3216c9bf3c2d93ee00bdd837769776554cdb41b79f62affcfd78bf657941752a"}
Sep 10 12:41:46.291479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:46.291440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9" event={"ID":"826a241c-f0cf-4722-84be-292191eca9c2","Type":"ContainerStarted","Data":"fcd039667098ecfdabe0625ce2e239e2474faf3def414d98995387ddf99e0841"}
Sep 10 12:41:46.312893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:46.312847    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9" podStartSLOduration=2.133856527 podStartE2EDuration="2.312832303s" totalImagesPullingTime="178.975776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:41:46.31162922 +0000 UTC m=+680.833798273" watchObservedRunningTime="2026-09-10 12:41:46.312832303 +0000 UTC m=+680.835001354"
Sep 10 12:41:47.733973 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:47.733940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:41:54.938173 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:54.938134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:54.938736 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:54.938190    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:54.950757 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:54.950733    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:41:55.345352 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:41:55.345326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9"
Sep 10 12:42:01.164931 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:42:01.164897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:42:07.155736 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:42:07.155701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:42:52.322412 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:42:52.322340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:42:52.429775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:42:52.429739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:28.233334 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.233294    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77f75b9d49-6zcpp"]
Sep 10 12:43:28.236124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.236106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.239792 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:43:28.239765    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-77f75b9d49-6zcpp\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="6868b0f8-2576-4b45-9881-d5f1ff7c5c1c" pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.264167 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.264144    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77f75b9d49-6zcpp"]
Sep 10 12:43:28.366533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.366503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6868b0f8-2576-4b45-9881-d5f1ff7c5c1c-tls-cert\") pod \"authorino-77f75b9d49-6zcpp\" (UID: \"6868b0f8-2576-4b45-9881-d5f1ff7c5c1c\") " pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.366703 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.366666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwqtr\" (UniqueName: \"kubernetes.io/projected/6868b0f8-2576-4b45-9881-d5f1ff7c5c1c-kube-api-access-fwqtr\") pod \"authorino-77f75b9d49-6zcpp\" (UID: \"6868b0f8-2576-4b45-9881-d5f1ff7c5c1c\") " pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.467895 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.467853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwqtr\" (UniqueName: \"kubernetes.io/projected/6868b0f8-2576-4b45-9881-d5f1ff7c5c1c-kube-api-access-fwqtr\") pod \"authorino-77f75b9d49-6zcpp\" (UID: \"6868b0f8-2576-4b45-9881-d5f1ff7c5c1c\") " pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.468101 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.467911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6868b0f8-2576-4b45-9881-d5f1ff7c5c1c-tls-cert\") pod \"authorino-77f75b9d49-6zcpp\" (UID: \"6868b0f8-2576-4b45-9881-d5f1ff7c5c1c\") " pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.470340 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.470310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6868b0f8-2576-4b45-9881-d5f1ff7c5c1c-tls-cert\") pod \"authorino-77f75b9d49-6zcpp\" (UID: \"6868b0f8-2576-4b45-9881-d5f1ff7c5c1c\") " pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.496010 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.495946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwqtr\" (UniqueName: \"kubernetes.io/projected/6868b0f8-2576-4b45-9881-d5f1ff7c5c1c-kube-api-access-fwqtr\") pod \"authorino-77f75b9d49-6zcpp\" (UID: \"6868b0f8-2576-4b45-9881-d5f1ff7c5c1c\") " pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.544826 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.544799    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f75b9d49-6zcpp"
Sep 10 12:43:28.818631 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:28.818588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77f75b9d49-6zcpp"]
Sep 10 12:43:28.820146 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:43:28.820120    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6868b0f8_2576_4b45_9881_d5f1ff7c5c1c.slice/crio-e390d426a346e031bf747c1784cd9a2d54357d87684872f352cf5063edaf1df3 WatchSource:0}: Error finding container e390d426a346e031bf747c1784cd9a2d54357d87684872f352cf5063edaf1df3: Status 404 returned error can't find the container with id e390d426a346e031bf747c1784cd9a2d54357d87684872f352cf5063edaf1df3
Sep 10 12:43:29.776353 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:29.776319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f75b9d49-6zcpp" event={"ID":"6868b0f8-2576-4b45-9881-d5f1ff7c5c1c","Type":"ContainerStarted","Data":"e390d426a346e031bf747c1784cd9a2d54357d87684872f352cf5063edaf1df3"}
Sep 10 12:43:30.783490 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:30.783453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f75b9d49-6zcpp" event={"ID":"6868b0f8-2576-4b45-9881-d5f1ff7c5c1c","Type":"ContainerStarted","Data":"efeffec649130d692f50bf30f08cdb5b5a65fb984c0d8dc24e452a57b6898797"}
Sep 10 12:43:30.831828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:30.831760    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77f75b9d49-6zcpp" podStartSLOduration=1.670685977 podStartE2EDuration="2.831744428s" totalImagesPullingTime="1.161058451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:43:30.831379395 +0000 UTC m=+785.353548448" watchObservedRunningTime="2026-09-10 12:43:30.831744428 +0000 UTC m=+785.353913480"
Sep 10 12:43:30.886526 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:30.886487    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79ff54bc7b-h6dcl"]
Sep 10 12:43:30.886786 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:30.886755    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl" podUID="67f43d6e-7a6b-4c86-b66a-30305f784362" containerName="authorino" containerID="cri-o://6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8" gracePeriod=30
Sep 10 12:43:31.141431 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.141405    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:43:31.190531 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.190490    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67f43d6e-7a6b-4c86-b66a-30305f784362-tls-cert\") pod \"67f43d6e-7a6b-4c86-b66a-30305f784362\" (UID: \"67f43d6e-7a6b-4c86-b66a-30305f784362\") "
Sep 10 12:43:31.190734 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.190562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97kk2\" (UniqueName: \"kubernetes.io/projected/67f43d6e-7a6b-4c86-b66a-30305f784362-kube-api-access-97kk2\") pod \"67f43d6e-7a6b-4c86-b66a-30305f784362\" (UID: \"67f43d6e-7a6b-4c86-b66a-30305f784362\") "
Sep 10 12:43:31.193781 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.193736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f43d6e-7a6b-4c86-b66a-30305f784362-kube-api-access-97kk2" (OuterVolumeSpecName: "kube-api-access-97kk2") pod "67f43d6e-7a6b-4c86-b66a-30305f784362" (UID: "67f43d6e-7a6b-4c86-b66a-30305f784362"). InnerVolumeSpecName "kube-api-access-97kk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:43:31.204324 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.204286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f43d6e-7a6b-4c86-b66a-30305f784362-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "67f43d6e-7a6b-4c86-b66a-30305f784362" (UID: "67f43d6e-7a6b-4c86-b66a-30305f784362"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:43:31.291518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.291483    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67f43d6e-7a6b-4c86-b66a-30305f784362-tls-cert\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:43:31.291518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.291512    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97kk2\" (UniqueName: \"kubernetes.io/projected/67f43d6e-7a6b-4c86-b66a-30305f784362-kube-api-access-97kk2\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:43:31.647460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.647412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:31.780010 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.779970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:31.789139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.789108    2577 generic.go:356] "Generic (PLEG): container finished" podID="67f43d6e-7a6b-4c86-b66a-30305f784362" containerID="6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8" exitCode=0
Sep 10 12:43:31.789510 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.789153    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl"
Sep 10 12:43:31.789510 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.789190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl" event={"ID":"67f43d6e-7a6b-4c86-b66a-30305f784362","Type":"ContainerDied","Data":"6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8"}
Sep 10 12:43:31.789510 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.789242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79ff54bc7b-h6dcl" event={"ID":"67f43d6e-7a6b-4c86-b66a-30305f784362","Type":"ContainerDied","Data":"0aba1af37d54aa573891053e97d5cd72bb8663d10a1da42b43b79f58f87dddf0"}
Sep 10 12:43:31.789510 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.789284    2577 scope.go:117] "RemoveContainer" containerID="6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8"
Sep 10 12:43:31.799133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.799110    2577 scope.go:117] "RemoveContainer" containerID="6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8"
Sep 10 12:43:31.799429 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:43:31.799409    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8\": container with ID starting with 6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8 not found: ID does not exist" containerID="6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8"
Sep 10 12:43:31.799507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.799439    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8"} err="failed to get container status \"6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8\": rpc error: code = NotFound desc = could not find container \"6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8\": container with ID starting with 6f4e954eb736d1a3a34c7bd0b4319e5330fd4f7d96d2b38b162dcda7c36cbda8 not found: ID does not exist"
Sep 10 12:43:31.846274 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.846187    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79ff54bc7b-h6dcl"]
Sep 10 12:43:31.851590 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:31.851562    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79ff54bc7b-h6dcl"]
Sep 10 12:43:32.000505 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:32.000427    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f43d6e-7a6b-4c86-b66a-30305f784362" path="/var/lib/kubelet/pods/67f43d6e-7a6b-4c86-b66a-30305f784362/volumes"
Sep 10 12:43:33.015417 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:33.015384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:33.131295 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:33.131265    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:41.830133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:41.830099    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:41.916024 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:41.915999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:53.166873 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:53.166799    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:53.276752 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:53.276574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:53.389880 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:53.389852    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:53.698373 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:53.698335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:53.959343 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:53.959269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:54.092357 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:54.092324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:54.189226 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:54.189191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:54.290548 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:54.290516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:54.380727 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:54.380691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:54.669162 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:54.669088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:54.773130 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:54.773101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:55.186178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.186144    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:55.267323 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.267291    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7716-8e6900"]
Sep 10 12:43:55.267699 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.267681    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f43d6e-7a6b-4c86-b66a-30305f784362" containerName="authorino"
Sep 10 12:43:55.267699 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.267692    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f43d6e-7a6b-4c86-b66a-30305f784362" containerName="authorino"
Sep 10 12:43:55.267772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.267748    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f43d6e-7a6b-4c86-b66a-30305f784362" containerName="authorino"
Sep 10 12:43:55.271083 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.271059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-8e6900"
Sep 10 12:43:55.275046 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.275029    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-w7tsg\""
Sep 10 12:43:55.281420 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.281396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7716-8e6900"]
Sep 10 12:43:55.392223 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.392192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4f6j\" (UniqueName: \"kubernetes.io/projected/6240d744-4e26-42a7-a50a-95a4d915eec7-kube-api-access-s4f6j\") pod \"test-curl-7716-8e6900\" (UID: \"6240d744-4e26-42a7-a50a-95a4d915eec7\") " pod="opendatahub/test-curl-7716-8e6900"
Sep 10 12:43:55.493516 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.493445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4f6j\" (UniqueName: \"kubernetes.io/projected/6240d744-4e26-42a7-a50a-95a4d915eec7-kube-api-access-s4f6j\") pod \"test-curl-7716-8e6900\" (UID: \"6240d744-4e26-42a7-a50a-95a4d915eec7\") " pod="opendatahub/test-curl-7716-8e6900"
Sep 10 12:43:55.510984 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.510960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4f6j\" (UniqueName: \"kubernetes.io/projected/6240d744-4e26-42a7-a50a-95a4d915eec7-kube-api-access-s4f6j\") pod \"test-curl-7716-8e6900\" (UID: \"6240d744-4e26-42a7-a50a-95a4d915eec7\") " pod="opendatahub/test-curl-7716-8e6900"
Sep 10 12:43:55.580912 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.580881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-8e6900"
Sep 10 12:43:55.858250 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.858228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7716-8e6900"]
Sep 10 12:43:55.860809 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:43:55.860771    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6240d744_4e26_42a7_a50a_95a4d915eec7.slice/crio-ec74dc0c5cc4f0f84026589bace2a8dc756930d511d09b4e7c0d13d7b9746083 WatchSource:0}: Error finding container ec74dc0c5cc4f0f84026589bace2a8dc756930d511d09b4e7c0d13d7b9746083: Status 404 returned error can't find the container with id ec74dc0c5cc4f0f84026589bace2a8dc756930d511d09b4e7c0d13d7b9746083
Sep 10 12:43:55.908805 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:55.908772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7716-8e6900" event={"ID":"6240d744-4e26-42a7-a50a-95a4d915eec7","Type":"ContainerStarted","Data":"ec74dc0c5cc4f0f84026589bace2a8dc756930d511d09b4e7c0d13d7b9746083"}
Sep 10 12:43:57.235535 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:57.235493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:57.340777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:57.340735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:57.462339 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:57.461023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:57.685213 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:57.685082    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:57.785640 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:57.785588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:58.743411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:58.743336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:58.843925 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:58.843899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:58.927043 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:58.927013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7716-8e6900" event={"ID":"6240d744-4e26-42a7-a50a-95a4d915eec7","Type":"ContainerStarted","Data":"ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e"}
Sep 10 12:43:59.072369 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.072338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:59.663981 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.663926    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7716-8e6900" podStartSLOduration=2.202275934 podStartE2EDuration="4.663908671s" totalImagesPullingTime="2.461632737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:43:58.953724011 +0000 UTC m=+813.475893065" watchObservedRunningTime="2026-09-10 12:43:59.663908671 +0000 UTC m=+814.186077722"
Sep 10 12:43:59.664587 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.664548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7716-8e6900"]
Sep 10 12:43:59.676413 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.676387    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7716-8e6900"]
Sep 10 12:43:59.741887 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.741855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:59.809007 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.808980    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7716-8977eb"]
Sep 10 12:43:59.809388 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.809371    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6240d744-4e26-42a7-a50a-95a4d915eec7" containerName="test-curl-7716-8e6900"
Sep 10 12:43:59.809430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.809391    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6240d744-4e26-42a7-a50a-95a4d915eec7" containerName="test-curl-7716-8e6900"
Sep 10 12:43:59.809474 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.809464    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6240d744-4e26-42a7-a50a-95a4d915eec7" containerName="test-curl-7716-8e6900"
Sep 10 12:43:59.812711 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.812686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:43:59.823971 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.823946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7716-8977eb"]
Sep 10 12:43:59.916324 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.916217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:43:59.937903 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:43:59.932958    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7716-8e6900" container="test-curl-7716-8e6900"
Sep 10 12:43:59.937903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:43:59.933561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqx4z\" (UniqueName: \"kubernetes.io/projected/b9838348-7077-4bb1-b342-4a7071029dfd-kube-api-access-sqx4z\") pod \"test-curl-7716-8977eb\" (UID: \"b9838348-7077-4bb1-b342-4a7071029dfd\") " pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:00.034582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.034540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx4z\" (UniqueName: \"kubernetes.io/projected/b9838348-7077-4bb1-b342-4a7071029dfd-kube-api-access-sqx4z\") pod \"test-curl-7716-8977eb\" (UID: \"b9838348-7077-4bb1-b342-4a7071029dfd\") " pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:00.066299 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.066260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqx4z\" (UniqueName: \"kubernetes.io/projected/b9838348-7077-4bb1-b342-4a7071029dfd-kube-api-access-sqx4z\") pod \"test-curl-7716-8977eb\" (UID: \"b9838348-7077-4bb1-b342-4a7071029dfd\") " pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:00.124037 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.123995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:00.194890 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.194860    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"]
Sep 10 12:44:00.199461 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.199442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.212232 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.212202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f85889-dockercfg-zw592\""
Sep 10 12:44:00.212232 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.212220    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f85889\""
Sep 10 12:44:00.308857 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.308822    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"]
Sep 10 12:44:00.312316 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.312296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:00.317566 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.317545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"]
Sep 10 12:44:00.330403 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.330380    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"]
Sep 10 12:44:00.338109 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.338087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86vn\" (UniqueName: \"kubernetes.io/projected/146a34bb-bfc3-4f38-8143-3840fb0983f2-kube-api-access-q86vn\") pod \"payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn\" (UID: \"146a34bb-bfc3-4f38-8143-3840fb0983f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.338199 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.338129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146a34bb-bfc3-4f38-8143-3840fb0983f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn\" (UID: \"146a34bb-bfc3-4f38-8143-3840fb0983f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.406063 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.406040    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7716-8977eb"]
Sep 10 12:44:00.406901 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:00.406874    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9838348_7077_4bb1_b342_4a7071029dfd.slice/crio-35eb297c8baf4455251aaccf84699a92ec85077c7634a379dd0daa78423191a9 WatchSource:0}: Error finding container 35eb297c8baf4455251aaccf84699a92ec85077c7634a379dd0daa78423191a9: Status 404 returned error can't find the container with id 35eb297c8baf4455251aaccf84699a92ec85077c7634a379dd0daa78423191a9
Sep 10 12:44:00.438546 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.438521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85729f7d-1f08-4334-982d-aa62ab456d55-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw\" (UID: \"85729f7d-1f08-4334-982d-aa62ab456d55\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:00.438645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.438555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146a34bb-bfc3-4f38-8143-3840fb0983f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn\" (UID: \"146a34bb-bfc3-4f38-8143-3840fb0983f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.438689 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.438666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5lr\" (UniqueName: \"kubernetes.io/projected/85729f7d-1f08-4334-982d-aa62ab456d55-kube-api-access-dh5lr\") pod \"payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw\" (UID: \"85729f7d-1f08-4334-982d-aa62ab456d55\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:00.438729 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.438713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86vn\" (UniqueName: \"kubernetes.io/projected/146a34bb-bfc3-4f38-8143-3840fb0983f2-kube-api-access-q86vn\") pod \"payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn\" (UID: \"146a34bb-bfc3-4f38-8143-3840fb0983f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.439257 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.439240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146a34bb-bfc3-4f38-8143-3840fb0983f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn\" (UID: \"146a34bb-bfc3-4f38-8143-3840fb0983f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.452075 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.452055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86vn\" (UniqueName: \"kubernetes.io/projected/146a34bb-bfc3-4f38-8143-3840fb0983f2-kube-api-access-q86vn\") pod \"payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn\" (UID: \"146a34bb-bfc3-4f38-8143-3840fb0983f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.510926 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.510900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.539694 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.539667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5lr\" (UniqueName: \"kubernetes.io/projected/85729f7d-1f08-4334-982d-aa62ab456d55-kube-api-access-dh5lr\") pod \"payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw\" (UID: \"85729f7d-1f08-4334-982d-aa62ab456d55\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:00.539826 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.539730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85729f7d-1f08-4334-982d-aa62ab456d55-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw\" (UID: \"85729f7d-1f08-4334-982d-aa62ab456d55\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:00.540428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.540408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85729f7d-1f08-4334-982d-aa62ab456d55-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw\" (UID: \"85729f7d-1f08-4334-982d-aa62ab456d55\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:00.553300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.553275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5lr\" (UniqueName: \"kubernetes.io/projected/85729f7d-1f08-4334-982d-aa62ab456d55-kube-api-access-dh5lr\") pod \"payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw\" (UID: \"85729f7d-1f08-4334-982d-aa62ab456d55\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:00.624029 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.623988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:00.789151 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.789112    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"]
Sep 10 12:44:00.790237 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:00.790210    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146a34bb_bfc3_4f38_8143_3840fb0983f2.slice/crio-cb30752048ce3eb669b9aabd515fbe680d135dd8f06cf1b2ad5600fa4ffd5527 WatchSource:0}: Error finding container cb30752048ce3eb669b9aabd515fbe680d135dd8f06cf1b2ad5600fa4ffd5527: Status 404 returned error can't find the container with id cb30752048ce3eb669b9aabd515fbe680d135dd8f06cf1b2ad5600fa4ffd5527
Sep 10 12:44:00.903804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.903782    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"]
Sep 10 12:44:00.905205 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:00.905183    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85729f7d_1f08_4334_982d_aa62ab456d55.slice/crio-e946340736a875baf66c259aabdbba7ff0ea2fb574bbba16f3f2b576069f107f WatchSource:0}: Error finding container e946340736a875baf66c259aabdbba7ff0ea2fb574bbba16f3f2b576069f107f: Status 404 returned error can't find the container with id e946340736a875baf66c259aabdbba7ff0ea2fb574bbba16f3f2b576069f107f
Sep 10 12:44:00.942997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.942237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn" event={"ID":"146a34bb-bfc3-4f38-8143-3840fb0983f2","Type":"ContainerStarted","Data":"1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8"}
Sep 10 12:44:00.942997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.942304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn" event={"ID":"146a34bb-bfc3-4f38-8143-3840fb0983f2","Type":"ContainerStarted","Data":"cb30752048ce3eb669b9aabd515fbe680d135dd8f06cf1b2ad5600fa4ffd5527"}
Sep 10 12:44:00.942997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.942488    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:00.944256 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.944232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw" event={"ID":"85729f7d-1f08-4334-982d-aa62ab456d55","Type":"ContainerStarted","Data":"e946340736a875baf66c259aabdbba7ff0ea2fb574bbba16f3f2b576069f107f"}
Sep 10 12:44:00.946416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.946390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7716-8977eb" event={"ID":"b9838348-7077-4bb1-b342-4a7071029dfd","Type":"ContainerStarted","Data":"85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6"}
Sep 10 12:44:00.946518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.946427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7716-8977eb" event={"ID":"b9838348-7077-4bb1-b342-4a7071029dfd","Type":"ContainerStarted","Data":"35eb297c8baf4455251aaccf84699a92ec85077c7634a379dd0daa78423191a9"}
Sep 10 12:44:00.946518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.946455    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7716-8e6900" podUID="6240d744-4e26-42a7-a50a-95a4d915eec7" containerName="test-curl-7716-8e6900" containerID="cri-o://ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e" gracePeriod=2
Sep 10 12:44:00.969904 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.969862    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn" podStartSLOduration=0.969847681 podStartE2EDuration="969.847681ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:00.968720195 +0000 UTC m=+815.490889248" watchObservedRunningTime="2026-09-10 12:44:00.969847681 +0000 UTC m=+815.492016797"
Sep 10 12:44:00.989972 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:00.989929    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7716-8977eb" podStartSLOduration=1.989916107 podStartE2EDuration="1.989916107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:00.987464265 +0000 UTC m=+815.509633316" watchObservedRunningTime="2026-09-10 12:44:00.989916107 +0000 UTC m=+815.512085158"
Sep 10 12:44:01.425567 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:01.425535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:01.546731 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:01.546696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:01.912314 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:01.912209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:01.953193 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:01.953166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw" event={"ID":"85729f7d-1f08-4334-982d-aa62ab456d55","Type":"ContainerStarted","Data":"47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911"}
Sep 10 12:44:01.953357 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:01.953230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:02.009112 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.009061    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw" podStartSLOduration=2.009044068 podStartE2EDuration="2.009044068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:02.005498733 +0000 UTC m=+816.527667784" watchObservedRunningTime="2026-09-10 12:44:02.009044068 +0000 UTC m=+816.531213121"
Sep 10 12:44:02.049147 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.049115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:02.079391 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.079369    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7716-8977eb"]
Sep 10 12:44:02.081645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.081621    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7716-8977eb"]
Sep 10 12:44:02.269669 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.269642    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"]
Sep 10 12:44:02.270050 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.270036    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9838348-7077-4bb1-b342-4a7071029dfd" containerName="test-curl-7716-8977eb"
Sep 10 12:44:02.270102 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.270052    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9838348-7077-4bb1-b342-4a7071029dfd" containerName="test-curl-7716-8977eb"
Sep 10 12:44:02.270140 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.270115    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9838348-7077-4bb1-b342-4a7071029dfd" containerName="test-curl-7716-8977eb"
Sep 10 12:44:02.273685 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.273664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:02.281716 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.281696    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-1312ea\""
Sep 10 12:44:02.295801 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.295761    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"]
Sep 10 12:44:02.297828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.297802    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"]
Sep 10 12:44:02.301723 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.301697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:02.328486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.328461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-1312ea-dockercfg-lfj9g\""
Sep 10 12:44:02.336611 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.336576    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"]
Sep 10 12:44:02.456797 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.456761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4\" (UID: \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:02.456943 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.456815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm88k\" (UniqueName: \"kubernetes.io/projected/1d31dcec-e6d9-4c53-a925-c05a14b67754-kube-api-access-tm88k\") pod \"payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l\" (UID: \"1d31dcec-e6d9-4c53-a925-c05a14b67754\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:02.456943 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.456871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr48x\" (UniqueName: \"kubernetes.io/projected/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-kube-api-access-cr48x\") pod \"payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4\" (UID: \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:02.457037 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.457004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d31dcec-e6d9-4c53-a925-c05a14b67754-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l\" (UID: \"1d31dcec-e6d9-4c53-a925-c05a14b67754\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:02.557953 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.557874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d31dcec-e6d9-4c53-a925-c05a14b67754-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l\" (UID: \"1d31dcec-e6d9-4c53-a925-c05a14b67754\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:02.557953 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.557932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4\" (UID: \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:02.558134 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.557976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm88k\" (UniqueName: \"kubernetes.io/projected/1d31dcec-e6d9-4c53-a925-c05a14b67754-kube-api-access-tm88k\") pod \"payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l\" (UID: \"1d31dcec-e6d9-4c53-a925-c05a14b67754\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:02.558134 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.557995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr48x\" (UniqueName: \"kubernetes.io/projected/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-kube-api-access-cr48x\") pod \"payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4\" (UID: \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:02.558771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.558750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4\" (UID: \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:02.558836 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.558779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d31dcec-e6d9-4c53-a925-c05a14b67754-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l\" (UID: \"1d31dcec-e6d9-4c53-a925-c05a14b67754\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:02.583313 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.583276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm88k\" (UniqueName: \"kubernetes.io/projected/1d31dcec-e6d9-4c53-a925-c05a14b67754-kube-api-access-tm88k\") pod \"payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l\" (UID: \"1d31dcec-e6d9-4c53-a925-c05a14b67754\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:02.590970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.590943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr48x\" (UniqueName: \"kubernetes.io/projected/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-kube-api-access-cr48x\") pod \"payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4\" (UID: \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:02.611664 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.611631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:02.696580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.696554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:02.885469 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.885381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:02.910450 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.910416    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"]
Sep 10 12:44:02.916074 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:02.916041    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d31dcec_e6d9_4c53_a925_c05a14b67754.slice/crio-323d11b8617735876e199dfe8dd7aa510771e1bd37140bacf83e920f7074e2cd WatchSource:0}: Error finding container 323d11b8617735876e199dfe8dd7aa510771e1bd37140bacf83e920f7074e2cd: Status 404 returned error can't find the container with id 323d11b8617735876e199dfe8dd7aa510771e1bd37140bacf83e920f7074e2cd
Sep 10 12:44:02.967098 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.966196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l" event={"ID":"1d31dcec-e6d9-4c53-a925-c05a14b67754","Type":"ContainerStarted","Data":"323d11b8617735876e199dfe8dd7aa510771e1bd37140bacf83e920f7074e2cd"}
Sep 10 12:44:02.967098 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:02.966499    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7716-8977eb" podUID="b9838348-7077-4bb1-b342-4a7071029dfd" containerName="test-curl-7716-8977eb" containerID="cri-o://85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6" gracePeriod=2
Sep 10 12:44:02.970108 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:44:02.969838    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7716-8977eb\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="b9838348-7077-4bb1-b342-4a7071029dfd" pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:03.076273 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.076239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:03.139720 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.139632    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-8e6900"
Sep 10 12:44:03.142739 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:44:03.142697    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7716-8977eb\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="b9838348-7077-4bb1-b342-4a7071029dfd" pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:03.193320 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.193294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"]
Sep 10 12:44:03.194772 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:03.194744    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8727ee6_1393_4331_b6b0_b7f5f3a25f39.slice/crio-5037d1d91e55e54520ca2fc1d6c2c3772a96be56c46c3b4554355440ef4f96f7 WatchSource:0}: Error finding container 5037d1d91e55e54520ca2fc1d6c2c3772a96be56c46c3b4554355440ef4f96f7: Status 404 returned error can't find the container with id 5037d1d91e55e54520ca2fc1d6c2c3772a96be56c46c3b4554355440ef4f96f7
Sep 10 12:44:03.209748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.209724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:03.265446 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.265420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4f6j\" (UniqueName: \"kubernetes.io/projected/6240d744-4e26-42a7-a50a-95a4d915eec7-kube-api-access-s4f6j\") pod \"6240d744-4e26-42a7-a50a-95a4d915eec7\" (UID: \"6240d744-4e26-42a7-a50a-95a4d915eec7\") "
Sep 10 12:44:03.267485 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.267459    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6240d744-4e26-42a7-a50a-95a4d915eec7-kube-api-access-s4f6j" (OuterVolumeSpecName: "kube-api-access-s4f6j") pod "6240d744-4e26-42a7-a50a-95a4d915eec7" (UID: "6240d744-4e26-42a7-a50a-95a4d915eec7"). InnerVolumeSpecName "kube-api-access-s4f6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:44:03.367191 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.367160    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4f6j\" (UniqueName: \"kubernetes.io/projected/6240d744-4e26-42a7-a50a-95a4d915eec7-kube-api-access-s4f6j\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:44:03.429959 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.429870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:03.702551 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.702466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:03.974229 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.974134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4" event={"ID":"d8727ee6-1393-4331-b6b0-b7f5f3a25f39","Type":"ContainerStarted","Data":"ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3"}
Sep 10 12:44:03.974229 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.974207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4" event={"ID":"d8727ee6-1393-4331-b6b0-b7f5f3a25f39","Type":"ContainerStarted","Data":"5037d1d91e55e54520ca2fc1d6c2c3772a96be56c46c3b4554355440ef4f96f7"}
Sep 10 12:44:03.974748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.974245    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:03.975992 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.975966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l" event={"ID":"1d31dcec-e6d9-4c53-a925-c05a14b67754","Type":"ContainerStarted","Data":"c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd"}
Sep 10 12:44:03.976133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.976075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:03.976553 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:44:03.976519    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7716-8977eb\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="b9838348-7077-4bb1-b342-4a7071029dfd" pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:03.977533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.977507    2577 generic.go:356] "Generic (PLEG): container finished" podID="6240d744-4e26-42a7-a50a-95a4d915eec7" containerID="ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e" exitCode=137
Sep 10 12:44:03.977656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.977535    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-8e6900"
Sep 10 12:44:03.977656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.977559    2577 scope.go:117] "RemoveContainer" containerID="ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e"
Sep 10 12:44:03.988944 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.988926    2577 scope.go:117] "RemoveContainer" containerID="ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e"
Sep 10 12:44:03.989386 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:44:03.989368    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e\": container with ID starting with ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e not found: ID does not exist" containerID="ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e"
Sep 10 12:44:03.989432 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:03.989394    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e"} err="failed to get container status \"ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e\": rpc error: code = NotFound desc = could not find container \"ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e\": container with ID starting with ca3ddc322989e91bd5f73327ca5d5f9eda3f1cc4e3cf015d6f96add5fb84768e not found: ID does not exist"
Sep 10 12:44:04.000271 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.000249    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6240d744-4e26-42a7-a50a-95a4d915eec7" path="/var/lib/kubelet/pods/6240d744-4e26-42a7-a50a-95a4d915eec7/volumes"
Sep 10 12:44:04.010893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.010851    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4" podStartSLOduration=2.01083772 podStartE2EDuration="2.01083772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:04.010371533 +0000 UTC m=+818.532540586" watchObservedRunningTime="2026-09-10 12:44:04.01083772 +0000 UTC m=+818.533006772"
Sep 10 12:44:04.023236 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.023211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:04.070535 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.070485    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l" podStartSLOduration=2.07046784 podStartE2EDuration="2.07046784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:04.056479891 +0000 UTC m=+818.578648941" watchObservedRunningTime="2026-09-10 12:44:04.07046784 +0000 UTC m=+818.592636893"
Sep 10 12:44:04.459784 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.459736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:04.930132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.930095    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"]
Sep 10 12:44:04.934785 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.934763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:04.938798 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.938774    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-30919a\""
Sep 10 12:44:04.973182 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:04.973156    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"]
Sep 10 12:44:05.082337 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.082300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced0f7d-b78b-4158-98ed-7242539fe3d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq\" (UID: \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:05.082759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.082433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2zs\" (UniqueName: \"kubernetes.io/projected/5ced0f7d-b78b-4158-98ed-7242539fe3d8-kube-api-access-nf2zs\") pod \"payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq\" (UID: \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:05.155987 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.155959    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:05.183284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.183210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced0f7d-b78b-4158-98ed-7242539fe3d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq\" (UID: \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:05.183284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.183283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2zs\" (UniqueName: \"kubernetes.io/projected/5ced0f7d-b78b-4158-98ed-7242539fe3d8-kube-api-access-nf2zs\") pod \"payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq\" (UID: \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:05.183975 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.183949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced0f7d-b78b-4158-98ed-7242539fe3d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq\" (UID: \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:05.197657 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.197634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2zs\" (UniqueName: \"kubernetes.io/projected/5ced0f7d-b78b-4158-98ed-7242539fe3d8-kube-api-access-nf2zs\") pod \"payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq\" (UID: \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:05.247823 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.247767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:05.284013 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.283948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqx4z\" (UniqueName: \"kubernetes.io/projected/b9838348-7077-4bb1-b342-4a7071029dfd-kube-api-access-sqx4z\") pod \"b9838348-7077-4bb1-b342-4a7071029dfd\" (UID: \"b9838348-7077-4bb1-b342-4a7071029dfd\") "
Sep 10 12:44:05.286409 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.286361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9838348-7077-4bb1-b342-4a7071029dfd-kube-api-access-sqx4z" (OuterVolumeSpecName: "kube-api-access-sqx4z") pod "b9838348-7077-4bb1-b342-4a7071029dfd" (UID: "b9838348-7077-4bb1-b342-4a7071029dfd"). InnerVolumeSpecName "kube-api-access-sqx4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:44:05.362175 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.362146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"]
Sep 10 12:44:05.366866 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.366846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:05.370729 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.370708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-30919a-dockercfg-whrmk\""
Sep 10 12:44:05.378561 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.378537    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"]
Sep 10 12:44:05.385719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.385696    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqx4z\" (UniqueName: \"kubernetes.io/projected/b9838348-7077-4bb1-b342-4a7071029dfd-kube-api-access-sqx4z\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:44:05.486744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.486655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc2b36-948b-468c-8629-89779664042f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r\" (UID: \"bbbc2b36-948b-468c-8629-89779664042f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:05.486744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.486696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzv7\" (UniqueName: \"kubernetes.io/projected/bbbc2b36-948b-468c-8629-89779664042f-kube-api-access-mnzv7\") pod \"payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r\" (UID: \"bbbc2b36-948b-468c-8629-89779664042f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:05.559077 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.559053    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"]
Sep 10 12:44:05.560091 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:05.560066    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ced0f7d_b78b_4158_98ed_7242539fe3d8.slice/crio-fe5cb9611e083e1a952361395805f53fcf4d2d5e58f80657aa6c93a63c0e116c WatchSource:0}: Error finding container fe5cb9611e083e1a952361395805f53fcf4d2d5e58f80657aa6c93a63c0e116c: Status 404 returned error can't find the container with id fe5cb9611e083e1a952361395805f53fcf4d2d5e58f80657aa6c93a63c0e116c
Sep 10 12:44:05.587911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.587878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzv7\" (UniqueName: \"kubernetes.io/projected/bbbc2b36-948b-468c-8629-89779664042f-kube-api-access-mnzv7\") pod \"payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r\" (UID: \"bbbc2b36-948b-468c-8629-89779664042f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:05.588040 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.588028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc2b36-948b-468c-8629-89779664042f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r\" (UID: \"bbbc2b36-948b-468c-8629-89779664042f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:05.588719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.588699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc2b36-948b-468c-8629-89779664042f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r\" (UID: \"bbbc2b36-948b-468c-8629-89779664042f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:05.611927 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.611899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzv7\" (UniqueName: \"kubernetes.io/projected/bbbc2b36-948b-468c-8629-89779664042f-kube-api-access-mnzv7\") pod \"payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r\" (UID: \"bbbc2b36-948b-468c-8629-89779664042f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:05.680965 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.680926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:05.717023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.716991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:05.991807 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.991778    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9838348-7077-4bb1-b342-4a7071029dfd" containerID="85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6" exitCode=137
Sep 10 12:44:05.991963 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.991830    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7716-8977eb"
Sep 10 12:44:05.991963 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.991864    2577 scope.go:117] "RemoveContainer" containerID="85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6"
Sep 10 12:44:05.993720 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:05.993694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq" event={"ID":"5ced0f7d-b78b-4158-98ed-7242539fe3d8","Type":"ContainerStarted","Data":"c4280ea389e159314cbf99d96349065a1e063c62c4af9cc42596aae83e22ba02"}
Sep 10 12:44:06.001711 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:06.001689    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9838348-7077-4bb1-b342-4a7071029dfd" path="/var/lib/kubelet/pods/b9838348-7077-4bb1-b342-4a7071029dfd/volumes"
Sep 10 12:44:06.002016 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:06.001997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq" event={"ID":"5ced0f7d-b78b-4158-98ed-7242539fe3d8","Type":"ContainerStarted","Data":"fe5cb9611e083e1a952361395805f53fcf4d2d5e58f80657aa6c93a63c0e116c"}
Sep 10 12:44:06.002076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:06.002051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:06.002136 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:06.002077    2577 scope.go:117] "RemoveContainer" containerID="85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6"
Sep 10 12:44:06.002390 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:44:06.002370    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6\": container with ID starting with 85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6 not found: ID does not exist" containerID="85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6"
Sep 10 12:44:06.002471 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:06.002401    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6"} err="failed to get container status \"85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6\": rpc error: code = NotFound desc = could not find container \"85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6\": container with ID starting with 85b2d53f7f23ebc1654ab5ad6381dac40042221619c7e0989b99f4b087a61df6 not found: ID does not exist"
Sep 10 12:44:06.026905 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:06.026880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbc2b36_948b_468c_8629_89779664042f.slice/crio-e841ed6675b84dc9410be88a0cd18a6c5df89f4edb643f271390db37f3ded94d WatchSource:0}: Error finding container e841ed6675b84dc9410be88a0cd18a6c5df89f4edb643f271390db37f3ded94d: Status 404 returned error can't find the container with id e841ed6675b84dc9410be88a0cd18a6c5df89f4edb643f271390db37f3ded94d
Sep 10 12:44:06.029247 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:06.028788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"]
Sep 10 12:44:06.128332 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:06.128271    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq" podStartSLOduration=2.128251665 podStartE2EDuration="2.128251665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:06.102195974 +0000 UTC m=+820.624365022" watchObservedRunningTime="2026-09-10 12:44:06.128251665 +0000 UTC m=+820.650420719"
Sep 10 12:44:07.002476 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:07.002442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r" event={"ID":"bbbc2b36-948b-468c-8629-89779664042f","Type":"ContainerStarted","Data":"c7cf659c87e9a256dee21f95df5a2ac183813002aeb4349066eede67bd0c69b0"}
Sep 10 12:44:07.002688 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:07.002535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r" event={"ID":"bbbc2b36-948b-468c-8629-89779664042f","Type":"ContainerStarted","Data":"e841ed6675b84dc9410be88a0cd18a6c5df89f4edb643f271390db37f3ded94d"}
Sep 10 12:44:07.002755 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:07.002706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:07.021134 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:07.021111    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:07.083487 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:07.083439    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r" podStartSLOduration=2.083423593 podStartE2EDuration="2.083423593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:07.070226588 +0000 UTC m=+821.592395640" watchObservedRunningTime="2026-09-10 12:44:07.083423593 +0000 UTC m=+821.605592645"
Sep 10 12:44:07.182342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:07.182282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:07.312082 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:07.312052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:08.590091 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:08.590045    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:09.657461 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:09.657429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:09.759116 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:09.759092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:10.005981 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:10.005885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:10.530098 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:10.530070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:10.661481 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:10.661444    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:11.233364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:11.233316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:11.956279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:11.956247    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:44:12.040410 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:12.040373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:12.509228 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:12.509177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:12.701520 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:12.701487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:12.838571 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:12.838544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:12.968959 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:12.968935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:44:13.349859 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:13.349822    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:13.560428 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:13.560395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:14.342323 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:14.342290    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:14.487079 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:14.487049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:14.987775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:14.987747    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:44:14.987922 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:14.987822    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:44:15.782311 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:15.782276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:15.974143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:15.974101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:16.208869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.208791    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:16.508895 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.508818    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"]
Sep 10 12:44:16.513564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.513547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:16.520073 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.520045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-5abff0-dockercfg-2rpzp\""
Sep 10 12:44:16.520073 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.520062    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5abff0\""
Sep 10 12:44:16.550283 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.550242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"]
Sep 10 12:44:16.578675 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.578649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9746030-7804-466a-b5dd-e15ade63c7ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz\" (UID: \"f9746030-7804-466a-b5dd-e15ade63c7ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:16.578790 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.578715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrn9c\" (UniqueName: \"kubernetes.io/projected/f9746030-7804-466a-b5dd-e15ade63c7ce-kube-api-access-hrn9c\") pod \"payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz\" (UID: \"f9746030-7804-466a-b5dd-e15ade63c7ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:16.600554 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.600529    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"]
Sep 10 12:44:16.604168 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.604150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:16.631864 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.631840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"]
Sep 10 12:44:16.679695 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.679672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9746030-7804-466a-b5dd-e15ade63c7ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz\" (UID: \"f9746030-7804-466a-b5dd-e15ade63c7ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:16.679813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.679721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn9c\" (UniqueName: \"kubernetes.io/projected/f9746030-7804-466a-b5dd-e15ade63c7ce-kube-api-access-hrn9c\") pod \"payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz\" (UID: \"f9746030-7804-466a-b5dd-e15ade63c7ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:16.679887 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.679838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d27f86-9097-4b0b-90ca-92ead9f5c537-plugins-config-volume\") pod \"payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv\" (UID: \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\") " pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:16.679887 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.679875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg45r\" (UniqueName: \"kubernetes.io/projected/b8d27f86-9097-4b0b-90ca-92ead9f5c537-kube-api-access-kg45r\") pod \"payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv\" (UID: \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\") " pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:16.680373 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.680350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9746030-7804-466a-b5dd-e15ade63c7ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz\" (UID: \"f9746030-7804-466a-b5dd-e15ade63c7ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:16.715026 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.714999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrn9c\" (UniqueName: \"kubernetes.io/projected/f9746030-7804-466a-b5dd-e15ade63c7ce-kube-api-access-hrn9c\") pod \"payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz\" (UID: \"f9746030-7804-466a-b5dd-e15ade63c7ce\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:16.781171 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.781142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d27f86-9097-4b0b-90ca-92ead9f5c537-plugins-config-volume\") pod \"payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv\" (UID: \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\") " pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:16.781294 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.781180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg45r\" (UniqueName: \"kubernetes.io/projected/b8d27f86-9097-4b0b-90ca-92ead9f5c537-kube-api-access-kg45r\") pod \"payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv\" (UID: \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\") " pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:16.781911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.781893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d27f86-9097-4b0b-90ca-92ead9f5c537-plugins-config-volume\") pod \"payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv\" (UID: \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\") " pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:16.823775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.823750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:16.833861 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.833836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg45r\" (UniqueName: \"kubernetes.io/projected/b8d27f86-9097-4b0b-90ca-92ead9f5c537-kube-api-access-kg45r\") pod \"payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv\" (UID: \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\") " pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:16.914203 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:16.914172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:17.120015 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:17.119988    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"]
Sep 10 12:44:17.122522 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:17.122497    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9746030_7804_466a_b5dd_e15ade63c7ce.slice/crio-03df7f802b335adc5ddf6be471e49c32735d4b7498800f053ba7af8bd0143bba WatchSource:0}: Error finding container 03df7f802b335adc5ddf6be471e49c32735d4b7498800f053ba7af8bd0143bba: Status 404 returned error can't find the container with id 03df7f802b335adc5ddf6be471e49c32735d4b7498800f053ba7af8bd0143bba
Sep 10 12:44:17.212712 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:17.212652    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"]
Sep 10 12:44:17.214726 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:17.214704    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d27f86_9097_4b0b_90ca_92ead9f5c537.slice/crio-6704826d3ddda1ac775fdd67f4f285d21a27c051f646c68aa8c0e86be3471750 WatchSource:0}: Error finding container 6704826d3ddda1ac775fdd67f4f285d21a27c051f646c68aa8c0e86be3471750: Status 404 returned error can't find the container with id 6704826d3ddda1ac775fdd67f4f285d21a27c051f646c68aa8c0e86be3471750
Sep 10 12:44:17.505492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:17.505409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:17.657233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:17.657201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:17.956268 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:17.956219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:18.008731 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.008706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:44:18.008879 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.008769    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:44:18.065633 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.065579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv" event={"ID":"b8d27f86-9097-4b0b-90ca-92ead9f5c537","Type":"ContainerStarted","Data":"60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018"}
Sep 10 12:44:18.065883 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.065861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv" event={"ID":"b8d27f86-9097-4b0b-90ca-92ead9f5c537","Type":"ContainerStarted","Data":"6704826d3ddda1ac775fdd67f4f285d21a27c051f646c68aa8c0e86be3471750"}
Sep 10 12:44:18.066332 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.066311    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:18.067998 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.067976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz" event={"ID":"f9746030-7804-466a-b5dd-e15ade63c7ce","Type":"ContainerStarted","Data":"75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c"}
Sep 10 12:44:18.068153 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.068137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz" event={"ID":"f9746030-7804-466a-b5dd-e15ade63c7ce","Type":"ContainerStarted","Data":"03df7f802b335adc5ddf6be471e49c32735d4b7498800f053ba7af8bd0143bba"}
Sep 10 12:44:18.068278 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.068266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:18.188010 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.187957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:18.323021 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.322960    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv" podStartSLOduration=2.322939095 podStartE2EDuration="2.322939095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:18.312781345 +0000 UTC m=+832.834950397" watchObservedRunningTime="2026-09-10 12:44:18.322939095 +0000 UTC m=+832.845108159"
Sep 10 12:44:18.400051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:18.400021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:20.176536 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:20.176511    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:20.399153 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:20.399116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:21.196195 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:21.196157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:21.639583 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:21.639550    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:21.837554 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:21.837520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:23.007926 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:23.007892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:24.655754 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:24.655715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:25.364612 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:25.364578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:26.303705 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:26.303673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:26.846945 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:26.846899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:28.022686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:28.022655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:29.075060 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:29.075027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:44:29.118921 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:29.118881    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz" podStartSLOduration=13.118866644 podStartE2EDuration="13.118866644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:18.431380732 +0000 UTC m=+832.953549786" watchObservedRunningTime="2026-09-10 12:44:29.118866644 +0000 UTC m=+843.641035695"
Sep 10 12:44:30.082333 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:30.082302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:44:30.348998 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:30.348925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:31.275959 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:31.275923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:32.482091 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:32.482057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:32.641914 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:32.641879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:32.949607 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:32.949574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:33.707172 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:33.707144    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:34.141471 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:34.141437    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:34.282753 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:34.282723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:34.390705 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:34.390675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:34.873997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:34.873966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:37.058425 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:37.058386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:37.224980 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:37.224947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:37.956830 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:37.956790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:40.059183 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:40.059147    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:40.168986 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:40.168959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:40.685276 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:40.685241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:41.423708 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.423679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:41.623271 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.623240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:41.827378 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.827347    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"]
Sep 10 12:44:41.831145 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.831129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:41.840764 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:44:41.840729    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:41.841921 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.841906    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-714b74-dockercfg-xhjwp\""
Sep 10 12:44:41.842759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.842741    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-714b74\""
Sep 10 12:44:41.860957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.860935    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"]
Sep 10 12:44:41.911921 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.911893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n\" (UID: \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:41.912048 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:41.912027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrqzm\" (UniqueName: \"kubernetes.io/projected/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-kube-api-access-vrqzm\") pod \"payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n\" (UID: \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:42.012802 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.012773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrqzm\" (UniqueName: \"kubernetes.io/projected/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-kube-api-access-vrqzm\") pod \"payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n\" (UID: \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:42.012982 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.012837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n\" (UID: \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:42.013436 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.013417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n\" (UID: \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:42.031629 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.031583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrqzm\" (UniqueName: \"kubernetes.io/projected/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-kube-api-access-vrqzm\") pod \"payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n\" (UID: \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:42.077263 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.077238    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"]
Sep 10 12:44:42.080919 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.080881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:42.098790 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.098769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"]
Sep 10 12:44:42.140710 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.140688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:42.214535 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.214486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk94p\" (UniqueName: \"kubernetes.io/projected/67869779-0a1b-455e-bea2-d96fec6c268e-kube-api-access-vk94p\") pod \"payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x\" (UID: \"67869779-0a1b-455e-bea2-d96fec6c268e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:42.214707 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.214617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67869779-0a1b-455e-bea2-d96fec6c268e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x\" (UID: \"67869779-0a1b-455e-bea2-d96fec6c268e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:42.315275 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.315240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67869779-0a1b-455e-bea2-d96fec6c268e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x\" (UID: \"67869779-0a1b-455e-bea2-d96fec6c268e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:42.315422 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.315338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk94p\" (UniqueName: \"kubernetes.io/projected/67869779-0a1b-455e-bea2-d96fec6c268e-kube-api-access-vk94p\") pod \"payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x\" (UID: \"67869779-0a1b-455e-bea2-d96fec6c268e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:42.316048 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.316030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67869779-0a1b-455e-bea2-d96fec6c268e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x\" (UID: \"67869779-0a1b-455e-bea2-d96fec6c268e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:42.330958 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.330910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk94p\" (UniqueName: \"kubernetes.io/projected/67869779-0a1b-455e-bea2-d96fec6c268e-kube-api-access-vk94p\") pod \"payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x\" (UID: \"67869779-0a1b-455e-bea2-d96fec6c268e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:42.391755 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.391730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:42.433298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.433271    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"]
Sep 10 12:44:42.436392 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:42.436359    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7d1cf6_d18f_40ad_a6a3_1188c3350da1.slice/crio-e041b6f62055996bd1b5cac7a6f212902446a1a7e8944cea62834bc024c5536b WatchSource:0}: Error finding container e041b6f62055996bd1b5cac7a6f212902446a1a7e8944cea62834bc024c5536b: Status 404 returned error can't find the container with id e041b6f62055996bd1b5cac7a6f212902446a1a7e8944cea62834bc024c5536b
Sep 10 12:44:42.683894 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:42.683851    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"]
Sep 10 12:44:42.686564 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:44:42.686530    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67869779_0a1b_455e_bea2_d96fec6c268e.slice/crio-58ab1e86cf9c1471ecc297270267cbd7dadecef68fee5a1a8487a4fcd81eacb5 WatchSource:0}: Error finding container 58ab1e86cf9c1471ecc297270267cbd7dadecef68fee5a1a8487a4fcd81eacb5: Status 404 returned error can't find the container with id 58ab1e86cf9c1471ecc297270267cbd7dadecef68fee5a1a8487a4fcd81eacb5
Sep 10 12:44:43.204053 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.204014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x" event={"ID":"67869779-0a1b-455e-bea2-d96fec6c268e","Type":"ContainerStarted","Data":"9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60"}
Sep 10 12:44:43.204229 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.204090    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:43.204229 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.204121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x" event={"ID":"67869779-0a1b-455e-bea2-d96fec6c268e","Type":"ContainerStarted","Data":"58ab1e86cf9c1471ecc297270267cbd7dadecef68fee5a1a8487a4fcd81eacb5"}
Sep 10 12:44:43.205567 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.205543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n" event={"ID":"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1","Type":"ContainerStarted","Data":"b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f"}
Sep 10 12:44:43.205697 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.205589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n" event={"ID":"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1","Type":"ContainerStarted","Data":"e041b6f62055996bd1b5cac7a6f212902446a1a7e8944cea62834bc024c5536b"}
Sep 10 12:44:43.205697 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.205627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:43.237004 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.236957    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x" podStartSLOduration=1.236937853 podStartE2EDuration="1.236937853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:43.236271114 +0000 UTC m=+857.758440186" watchObservedRunningTime="2026-09-10 12:44:43.236937853 +0000 UTC m=+857.759106905"
Sep 10 12:44:43.272736 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.272697    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n" podStartSLOduration=2.272683056 podStartE2EDuration="2.272683056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:43.27193562 +0000 UTC m=+857.794104682" watchObservedRunningTime="2026-09-10 12:44:43.272683056 +0000 UTC m=+857.794852108"
Sep 10 12:44:43.471061 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:43.470972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:45.633631 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:45.633587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:45.785069 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:45.785019    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:45.923377 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:45.923295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:46.199111 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:46.199023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:53.032611 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:53.032556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:53.102161 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:53.102131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:54.213467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:54.213433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:44:54.213864 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:54.213507    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:44:59.526767 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:59.526732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:59.657130 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:59.657093    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:59.760221 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:59.760187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:44:59.860384 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:44:59.860303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:00.169441 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.169352    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"]
Sep 10 12:45:00.172921 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.172896    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"]
Sep 10 12:45:00.173050 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.173005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"
Sep 10 12:45:00.176366 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.176338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"
Sep 10 12:45:00.178701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.178670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:45:00.184325 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.184308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"
Sep 10 12:45:00.190485 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.190470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"
Sep 10 12:45:00.206150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.206119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"]
Sep 10 12:45:00.208923 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.208896    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"]
Sep 10 12:45:00.229319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.229278    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"]
Sep 10 12:45:00.234886 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.234860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"
Sep 10 12:45:00.249060 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.249038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"
Sep 10 12:45:00.267867 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.266197    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"]
Sep 10 12:45:00.488298 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.488274    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"]
Sep 10 12:45:00.490271 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:45:00.490241    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24409d2_4fa9_4caf_bca3_710bb9539119.slice/crio-107ad3eca00feee6cb724295382e567b31a8e238250cd2c591b0c3f40a6793ae WatchSource:0}: Error finding container 107ad3eca00feee6cb724295382e567b31a8e238250cd2c591b0c3f40a6793ae: Status 404 returned error can't find the container with id 107ad3eca00feee6cb724295382e567b31a8e238250cd2c591b0c3f40a6793ae
Sep 10 12:45:00.497914 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.497888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"]
Sep 10 12:45:00.499028 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:45:00.498998    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode044bb97_066e_4aae_886c_b9790dfd6bff.slice/crio-184b8abceaa5c3e8e92872fdfc8e7394f567e4ea4f73b93fa8191e57da43f797 WatchSource:0}: Error finding container 184b8abceaa5c3e8e92872fdfc8e7394f567e4ea4f73b93fa8191e57da43f797: Status 404 returned error can't find the container with id 184b8abceaa5c3e8e92872fdfc8e7394f567e4ea4f73b93fa8191e57da43f797
Sep 10 12:45:00.567547 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.567519    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"]
Sep 10 12:45:00.568952 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:45:00.568926    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22558955_341f_4c29_968f_7c980369f3a8.slice/crio-9f4c9094914b31ffd29f897d691411fadb51e80b7ab04ec8e440e979c3be37b3 WatchSource:0}: Error finding container 9f4c9094914b31ffd29f897d691411fadb51e80b7ab04ec8e440e979c3be37b3: Status 404 returned error can't find the container with id 9f4c9094914b31ffd29f897d691411fadb51e80b7ab04ec8e440e979c3be37b3
Sep 10 12:45:00.683649 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.683589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:00.774017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:00.773987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:01.020031 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:01.019983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:01.168069 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:01.167995    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:01.301835 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:01.301803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np" event={"ID":"22558955-341f-4c29-968f-7c980369f3a8","Type":"ContainerStarted","Data":"9f4c9094914b31ffd29f897d691411fadb51e80b7ab04ec8e440e979c3be37b3"}
Sep 10 12:45:01.303036 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:01.303002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp" event={"ID":"f24409d2-4fa9-4caf-bca3-710bb9539119","Type":"ContainerStarted","Data":"107ad3eca00feee6cb724295382e567b31a8e238250cd2c591b0c3f40a6793ae"}
Sep 10 12:45:01.304292 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:01.304273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb" event={"ID":"e044bb97-066e-4aae-886c-b9790dfd6bff","Type":"ContainerStarted","Data":"184b8abceaa5c3e8e92872fdfc8e7394f567e4ea4f73b93fa8191e57da43f797"}
Sep 10 12:45:02.400572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:02.400473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:03.317539 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:03.317441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np" event={"ID":"22558955-341f-4c29-968f-7c980369f3a8","Type":"ContainerStarted","Data":"74b31bb1cbbf31cbe418135d984234ae2e3e7e8e4793017c20207e0d7928a935"}
Sep 10 12:45:03.325742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:03.325682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb" event={"ID":"e044bb97-066e-4aae-886c-b9790dfd6bff","Type":"ContainerStarted","Data":"63db7dc82af5086582c6b7d31d29e0811a270b7c6887a032a48119ca20f9fad4"}
Sep 10 12:45:03.342280 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:03.342234    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np" podStartSLOduration=1.13121126 podStartE2EDuration="3.342221582s" totalImagesPullingTime="2.211010322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:03.340878381 +0000 UTC m=+877.863047434" watchObservedRunningTime="2026-09-10 12:45:03.342221582 +0000 UTC m=+877.864390686"
Sep 10 12:45:03.367893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:03.367836    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb" podStartSLOduration=1.092486235 podStartE2EDuration="3.36781905s" totalImagesPullingTime="2.275332815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:03.365831203 +0000 UTC m=+877.888000258" watchObservedRunningTime="2026-09-10 12:45:03.36781905 +0000 UTC m=+877.889988124"
Sep 10 12:45:03.660261 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:03.660141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:03.761151 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:03.761124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:03.963173 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:03.963096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:04.060143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.060112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:04.162193 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.162161    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:04.332891 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.332861    2577 generic.go:356] "Generic (PLEG): container finished" podID="22558955-341f-4c29-968f-7c980369f3a8" containerID="74b31bb1cbbf31cbe418135d984234ae2e3e7e8e4793017c20207e0d7928a935" exitCode=0
Sep 10 12:45:04.333068 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.332953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np" event={"ID":"22558955-341f-4c29-968f-7c980369f3a8","Type":"ContainerDied","Data":"74b31bb1cbbf31cbe418135d984234ae2e3e7e8e4793017c20207e0d7928a935"}
Sep 10 12:45:04.334400 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.334376    2577 generic.go:356] "Generic (PLEG): container finished" podID="e044bb97-066e-4aae-886c-b9790dfd6bff" containerID="63db7dc82af5086582c6b7d31d29e0811a270b7c6887a032a48119ca20f9fad4" exitCode=0
Sep 10 12:45:04.334507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.334444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb" event={"ID":"e044bb97-066e-4aae-886c-b9790dfd6bff","Type":"ContainerDied","Data":"63db7dc82af5086582c6b7d31d29e0811a270b7c6887a032a48119ca20f9fad4"}
Sep 10 12:45:04.614918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.614843    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:04.742959 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.742928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:04.789271 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.789231    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"]
Sep 10 12:45:04.792927 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.792904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:04.795942 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.795924    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-760a0d05\""
Sep 10 12:45:04.804820 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.804798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"]
Sep 10 12:45:04.824525 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.824503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt82\" (UniqueName: \"kubernetes.io/projected/4244451a-8caa-4c35-ad8f-8225447a18a8-kube-api-access-5dt82\") pod \"payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6\" (UID: \"4244451a-8caa-4c35-ad8f-8225447a18a8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:04.824657 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.824540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4244451a-8caa-4c35-ad8f-8225447a18a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6\" (UID: \"4244451a-8caa-4c35-ad8f-8225447a18a8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:04.926007 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.925925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt82\" (UniqueName: \"kubernetes.io/projected/4244451a-8caa-4c35-ad8f-8225447a18a8-kube-api-access-5dt82\") pod \"payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6\" (UID: \"4244451a-8caa-4c35-ad8f-8225447a18a8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:04.926007 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.925979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4244451a-8caa-4c35-ad8f-8225447a18a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6\" (UID: \"4244451a-8caa-4c35-ad8f-8225447a18a8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:04.926732 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.926710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4244451a-8caa-4c35-ad8f-8225447a18a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6\" (UID: \"4244451a-8caa-4c35-ad8f-8225447a18a8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:04.935145 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:04.935122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt82\" (UniqueName: \"kubernetes.io/projected/4244451a-8caa-4c35-ad8f-8225447a18a8-kube-api-access-5dt82\") pod \"payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6\" (UID: \"4244451a-8caa-4c35-ad8f-8225447a18a8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:05.103892 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:05.103857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:05.191187 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:05.191116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:05.387691 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:05.387632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"]
Sep 10 12:45:05.481303 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:05.481261    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"
Sep 10 12:45:05.490290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:05.490267    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"
Sep 10 12:45:06.347660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.347630    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"
Sep 10 12:45:06.347660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.347646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np" event={"ID":"22558955-341f-4c29-968f-7c980369f3a8","Type":"ContainerDied","Data":"9f4c9094914b31ffd29f897d691411fadb51e80b7ab04ec8e440e979c3be37b3"}
Sep 10 12:45:06.348165 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.347698    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f4c9094914b31ffd29f897d691411fadb51e80b7ab04ec8e440e979c3be37b3"
Sep 10 12:45:06.349414 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.349389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6" event={"ID":"4244451a-8caa-4c35-ad8f-8225447a18a8","Type":"ContainerStarted","Data":"63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab"}
Sep 10 12:45:06.349529 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.349432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6" event={"ID":"4244451a-8caa-4c35-ad8f-8225447a18a8","Type":"ContainerStarted","Data":"26dcd3a971e3be6a66523996b6d3e715f3279a6af82aa039b9bd5978dab97cb3"}
Sep 10 12:45:06.349630 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.349551    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:06.350952 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.350937    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"
Sep 10 12:45:06.351052 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.350967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb" event={"ID":"e044bb97-066e-4aae-886c-b9790dfd6bff","Type":"ContainerDied","Data":"184b8abceaa5c3e8e92872fdfc8e7394f567e4ea4f73b93fa8191e57da43f797"}
Sep 10 12:45:06.351052 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.351002    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="184b8abceaa5c3e8e92872fdfc8e7394f567e4ea4f73b93fa8191e57da43f797"
Sep 10 12:45:06.397562 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.397520    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6" podStartSLOduration=2.3975049889999998 podStartE2EDuration="2.397504989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:06.387376723 +0000 UTC m=+880.909545776" watchObservedRunningTime="2026-09-10 12:45:06.397504989 +0000 UTC m=+880.919674032"
Sep 10 12:45:06.751737 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:06.751644    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:07.040491 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:07.040447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:17.359761 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:17.359727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:18.887647 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:18.887545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:23.453829 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:23.453731    2577 generic.go:356] "Generic (PLEG): container finished" podID="f24409d2-4fa9-4caf-bca3-710bb9539119" containerID="ba4159a16f7a216d64682076c77b21ef0deebe6fcc0bd951a2ac2472f2e726b8" exitCode=0
Sep 10 12:45:23.453829 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:23.453792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp" event={"ID":"f24409d2-4fa9-4caf-bca3-710bb9539119","Type":"ContainerDied","Data":"ba4159a16f7a216d64682076c77b21ef0deebe6fcc0bd951a2ac2472f2e726b8"}
Sep 10 12:45:24.591270 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:24.591246    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"
Sep 10 12:45:25.466145 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:25.466115    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"
Sep 10 12:45:25.466316 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:25.466111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp" event={"ID":"f24409d2-4fa9-4caf-bca3-710bb9539119","Type":"ContainerDied","Data":"107ad3eca00feee6cb724295382e567b31a8e238250cd2c591b0c3f40a6793ae"}
Sep 10 12:45:25.466316 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:25.466237    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="107ad3eca00feee6cb724295382e567b31a8e238250cd2c591b0c3f40a6793ae"
Sep 10 12:45:25.934496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:25.934463    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:45:25.935409 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:25.935387    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:45:25.939138 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:25.939104    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:45:25.939742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:25.939721    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:45:27.345179 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:27.345144    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:28.580861 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.580827    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"]
Sep 10 12:45:28.581529 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581502    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e044bb97-066e-4aae-886c-b9790dfd6bff" containerName="cleanup"
Sep 10 12:45:28.581529 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581531    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e044bb97-066e-4aae-886c-b9790dfd6bff" containerName="cleanup"
Sep 10 12:45:28.581744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581558    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22558955-341f-4c29-968f-7c980369f3a8" containerName="cleanup"
Sep 10 12:45:28.581744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581565    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="22558955-341f-4c29-968f-7c980369f3a8" containerName="cleanup"
Sep 10 12:45:28.581744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581573    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f24409d2-4fa9-4caf-bca3-710bb9539119" containerName="cleanup"
Sep 10 12:45:28.581744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581579    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f24409d2-4fa9-4caf-bca3-710bb9539119" containerName="cleanup"
Sep 10 12:45:28.581744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581679    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="22558955-341f-4c29-968f-7c980369f3a8" containerName="cleanup"
Sep 10 12:45:28.581744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581690    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e044bb97-066e-4aae-886c-b9790dfd6bff" containerName="cleanup"
Sep 10 12:45:28.581744 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.581702    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f24409d2-4fa9-4caf-bca3-710bb9539119" containerName="cleanup"
Sep 10 12:45:28.585054 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.585034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"
Sep 10 12:45:28.589047 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.589026    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:45:28.597391 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.597369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"]
Sep 10 12:45:28.751582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.751548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba56bb-9c65-457c-af4b-ee667ae418c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd\" (UID: \"a8ba56bb-9c65-457c-af4b-ee667ae418c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"
Sep 10 12:45:28.852456 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.852375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba56bb-9c65-457c-af4b-ee667ae418c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd\" (UID: \"a8ba56bb-9c65-457c-af4b-ee667ae418c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"
Sep 10 12:45:28.853019 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.852998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba56bb-9c65-457c-af4b-ee667ae418c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd\" (UID: \"a8ba56bb-9c65-457c-af4b-ee667ae418c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"
Sep 10 12:45:28.895929 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:28.895901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"
Sep 10 12:45:29.168163 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:29.168138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"]
Sep 10 12:45:29.169949 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:45:29.169919    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ba56bb_9c65_457c_af4b_ee667ae418c4.slice/crio-36be0fb68d98244a7f8a6a85b1d012ab4f7f9c91228b598913cf7a0127b7d9be WatchSource:0}: Error finding container 36be0fb68d98244a7f8a6a85b1d012ab4f7f9c91228b598913cf7a0127b7d9be: Status 404 returned error can't find the container with id 36be0fb68d98244a7f8a6a85b1d012ab4f7f9c91228b598913cf7a0127b7d9be
Sep 10 12:45:29.489981 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:29.489949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd" event={"ID":"a8ba56bb-9c65-457c-af4b-ee667ae418c4","Type":"ContainerStarted","Data":"ae34edf81982e507ff96a255a87a21ccbc5b7f39f68f467551e6242c4f775fef"}
Sep 10 12:45:29.490086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:29.490004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd" event={"ID":"a8ba56bb-9c65-457c-af4b-ee667ae418c4","Type":"ContainerStarted","Data":"36be0fb68d98244a7f8a6a85b1d012ab4f7f9c91228b598913cf7a0127b7d9be"}
Sep 10 12:45:29.511911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:29.511866    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd" podStartSLOduration=1.5118517200000001 podStartE2EDuration="1.51185172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:29.510845083 +0000 UTC m=+904.033014135" watchObservedRunningTime="2026-09-10 12:45:29.51185172 +0000 UTC m=+904.034020818"
Sep 10 12:45:30.496310 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:30.496278    2577 generic.go:356] "Generic (PLEG): container finished" podID="a8ba56bb-9c65-457c-af4b-ee667ae418c4" containerID="ae34edf81982e507ff96a255a87a21ccbc5b7f39f68f467551e6242c4f775fef" exitCode=0
Sep 10 12:45:30.496702 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:30.496316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd" event={"ID":"a8ba56bb-9c65-457c-af4b-ee667ae418c4","Type":"ContainerDied","Data":"ae34edf81982e507ff96a255a87a21ccbc5b7f39f68f467551e6242c4f775fef"}
Sep 10 12:45:31.644609 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:31.644578    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"
Sep 10 12:45:31.779455 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:31.779427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba56bb-9c65-457c-af4b-ee667ae418c4-maas-api-service-ca\") pod \"a8ba56bb-9c65-457c-af4b-ee667ae418c4\" (UID: \"a8ba56bb-9c65-457c-af4b-ee667ae418c4\") "
Sep 10 12:45:31.779791 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:31.779765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ba56bb-9c65-457c-af4b-ee667ae418c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8ba56bb-9c65-457c-af4b-ee667ae418c4" (UID: "a8ba56bb-9c65-457c-af4b-ee667ae418c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:45:31.880634 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:31.880579    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ba56bb-9c65-457c-af4b-ee667ae418c4-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:45:32.514549 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:32.514518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"
Sep 10 12:45:32.514746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:32.514567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd" event={"ID":"a8ba56bb-9c65-457c-af4b-ee667ae418c4","Type":"ContainerDied","Data":"36be0fb68d98244a7f8a6a85b1d012ab4f7f9c91228b598913cf7a0127b7d9be"}
Sep 10 12:45:32.514746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:32.514650    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36be0fb68d98244a7f8a6a85b1d012ab4f7f9c91228b598913cf7a0127b7d9be"
Sep 10 12:45:35.402847 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.402814    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:35.422307 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.422276    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"]
Sep 10 12:45:35.422532 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.422485    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6" podUID="4244451a-8caa-4c35-ad8f-8225447a18a8" containerName="payload-pre-processing" containerID="cri-o://63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab" gracePeriod=30
Sep 10 12:45:35.502364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.502310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:35.606003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.605974    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:35.657699 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.657622    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"]
Sep 10 12:45:35.659902 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.659879    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-760a0d05-dsvbd"]
Sep 10 12:45:35.673887 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.673863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:35.685917 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.685898    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:35.817953 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.817921    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4244451a-8caa-4c35-ad8f-8225447a18a8-plugins-config-volume\") pod \"4244451a-8caa-4c35-ad8f-8225447a18a8\" (UID: \"4244451a-8caa-4c35-ad8f-8225447a18a8\") "
Sep 10 12:45:35.818146 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.818032    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dt82\" (UniqueName: \"kubernetes.io/projected/4244451a-8caa-4c35-ad8f-8225447a18a8-kube-api-access-5dt82\") pod \"4244451a-8caa-4c35-ad8f-8225447a18a8\" (UID: \"4244451a-8caa-4c35-ad8f-8225447a18a8\") "
Sep 10 12:45:35.818384 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.818354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4244451a-8caa-4c35-ad8f-8225447a18a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4244451a-8caa-4c35-ad8f-8225447a18a8" (UID: "4244451a-8caa-4c35-ad8f-8225447a18a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:45:35.820702 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.820667    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4244451a-8caa-4c35-ad8f-8225447a18a8-kube-api-access-5dt82" (OuterVolumeSpecName: "kube-api-access-5dt82") pod "4244451a-8caa-4c35-ad8f-8225447a18a8" (UID: "4244451a-8caa-4c35-ad8f-8225447a18a8"). InnerVolumeSpecName "kube-api-access-5dt82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:45:35.864232 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.864204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:35.919462 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.919393    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dt82\" (UniqueName: \"kubernetes.io/projected/4244451a-8caa-4c35-ad8f-8225447a18a8-kube-api-access-5dt82\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:45:35.919462 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:35.919421    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4244451a-8caa-4c35-ad8f-8225447a18a8-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:45:36.001588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.001557    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8ba56bb-9c65-457c-af4b-ee667ae418c4" path="/var/lib/kubelet/pods/a8ba56bb-9c65-457c-af4b-ee667ae418c4/volumes"
Sep 10 12:45:36.388866 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.388838    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:36.537664 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.537632    2577 generic.go:356] "Generic (PLEG): container finished" podID="4244451a-8caa-4c35-ad8f-8225447a18a8" containerID="63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab" exitCode=0
Sep 10 12:45:36.538119 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.537683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6" event={"ID":"4244451a-8caa-4c35-ad8f-8225447a18a8","Type":"ContainerDied","Data":"63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab"}
Sep 10 12:45:36.538119 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.537726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6" event={"ID":"4244451a-8caa-4c35-ad8f-8225447a18a8","Type":"ContainerDied","Data":"26dcd3a971e3be6a66523996b6d3e715f3279a6af82aa039b9bd5978dab97cb3"}
Sep 10 12:45:36.538119 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.537747    2577 scope.go:117] "RemoveContainer" containerID="63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab"
Sep 10 12:45:36.538119 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.537687    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"
Sep 10 12:45:36.547457 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.547440    2577 scope.go:117] "RemoveContainer" containerID="63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab"
Sep 10 12:45:36.547808 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:45:36.547777    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab\": container with ID starting with 63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab not found: ID does not exist" containerID="63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab"
Sep 10 12:45:36.547892 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.547815    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab"} err="failed to get container status \"63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab\": rpc error: code = NotFound desc = could not find container \"63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab\": container with ID starting with 63ee094c4e7b0fc599d43da168bbc11400cc2be0ae7fc30476319848ef5a2aab not found: ID does not exist"
Sep 10 12:45:36.583038 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.583017    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"]
Sep 10 12:45:36.588639 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.588622    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-760a0d05-6fbb9b8b78-d9rn6"]
Sep 10 12:45:36.754571 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:36.754501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:37.142025 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:37.141988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:38.001394 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:38.001351    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4244451a-8caa-4c35-ad8f-8225447a18a8" path="/var/lib/kubelet/pods/4244451a-8caa-4c35-ad8f-8225447a18a8/volumes"
Sep 10 12:45:39.956997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:39.956964    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:40.037836 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:40.037803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:41.161895 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.161863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:41.265175 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.265132    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:41.401213 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.401181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:41.878650 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.878621    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:41.994369 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.994335    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"]
Sep 10 12:45:41.994970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.994947    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4244451a-8caa-4c35-ad8f-8225447a18a8" containerName="payload-pre-processing"
Sep 10 12:45:41.994970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.994971    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4244451a-8caa-4c35-ad8f-8225447a18a8" containerName="payload-pre-processing"
Sep 10 12:45:41.995121 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.994981    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8ba56bb-9c65-457c-af4b-ee667ae418c4" containerName="revoke-keys"
Sep 10 12:45:41.995121 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.994986    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8ba56bb-9c65-457c-af4b-ee667ae418c4" containerName="revoke-keys"
Sep 10 12:45:41.995121 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.995100    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8ba56bb-9c65-457c-af4b-ee667ae418c4" containerName="revoke-keys"
Sep 10 12:45:41.995121 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:41.995115    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4244451a-8caa-4c35-ad8f-8225447a18a8" containerName="payload-pre-processing"
Sep 10 12:45:42.000957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.000936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:42.001460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.001433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:42.003910 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.003892    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d7fcd5f6-dockercfg-tsv45\""
Sep 10 12:45:42.004295 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.004274    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d7fcd5f6\""
Sep 10 12:45:42.007831 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.007810    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"]
Sep 10 12:45:42.013297 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.013277    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"]
Sep 10 12:45:42.013385 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.013362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:42.015062 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.015043    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"]
Sep 10 12:45:42.074544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.074510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv69d\" (UniqueName: \"kubernetes.io/projected/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-kube-api-access-mv69d\") pod \"payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7\" (UID: \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:42.074748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.074648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7\" (UID: \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:42.074748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.074688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a7cd23-e87b-477a-a068-fd8a32eeec0b-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth\" (UID: \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\") " pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:42.074748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.074711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mv59\" (UniqueName: \"kubernetes.io/projected/67a7cd23-e87b-477a-a068-fd8a32eeec0b-kube-api-access-7mv59\") pod \"payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth\" (UID: \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\") " pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:42.175970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.175894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv69d\" (UniqueName: \"kubernetes.io/projected/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-kube-api-access-mv69d\") pod \"payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7\" (UID: \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:42.176338 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.176035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7\" (UID: \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:42.176338 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.176083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a7cd23-e87b-477a-a068-fd8a32eeec0b-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth\" (UID: \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\") " pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:42.176338 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.176105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mv59\" (UniqueName: \"kubernetes.io/projected/67a7cd23-e87b-477a-a068-fd8a32eeec0b-kube-api-access-7mv59\") pod \"payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth\" (UID: \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\") " pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:42.176817 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.176797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7\" (UID: \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:42.176860 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.176817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a7cd23-e87b-477a-a068-fd8a32eeec0b-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth\" (UID: \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\") " pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:42.188774 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.188742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv69d\" (UniqueName: \"kubernetes.io/projected/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-kube-api-access-mv69d\") pod \"payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7\" (UID: \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:42.188931 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.188781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mv59\" (UniqueName: \"kubernetes.io/projected/67a7cd23-e87b-477a-a068-fd8a32eeec0b-kube-api-access-7mv59\") pod \"payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth\" (UID: \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\") " pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:42.313106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.313072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:42.323568 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.323538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:42.608346 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.608320    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"]
Sep 10 12:45:42.612132 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:45:42.612041    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b2eb03_772f_42d7_8027_f0b4d08c3b7c.slice/crio-f58f09072457d16e26c608d0f745bfd1518abefaacfa4aef5c1b1bacc82075ea WatchSource:0}: Error finding container f58f09072457d16e26c608d0f745bfd1518abefaacfa4aef5c1b1bacc82075ea: Status 404 returned error can't find the container with id f58f09072457d16e26c608d0f745bfd1518abefaacfa4aef5c1b1bacc82075ea
Sep 10 12:45:42.636997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:42.636974    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"]
Sep 10 12:45:42.646730 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:45:42.646706    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a7cd23_e87b_477a_a068_fd8a32eeec0b.slice/crio-2a94899c5de95e501509a8ce302b9db33a97c409070a3222c5f033f474296140 WatchSource:0}: Error finding container 2a94899c5de95e501509a8ce302b9db33a97c409070a3222c5f033f474296140: Status 404 returned error can't find the container with id 2a94899c5de95e501509a8ce302b9db33a97c409070a3222c5f033f474296140
Sep 10 12:45:43.579824 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:43.579781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth" event={"ID":"67a7cd23-e87b-477a-a068-fd8a32eeec0b","Type":"ContainerStarted","Data":"ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d"}
Sep 10 12:45:43.580272 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:43.579850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth" event={"ID":"67a7cd23-e87b-477a-a068-fd8a32eeec0b","Type":"ContainerStarted","Data":"2a94899c5de95e501509a8ce302b9db33a97c409070a3222c5f033f474296140"}
Sep 10 12:45:43.580272 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:43.579973    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:45:43.581526 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:43.581488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7" event={"ID":"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c","Type":"ContainerStarted","Data":"9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359"}
Sep 10 12:45:43.581683 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:43.581530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7" event={"ID":"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c","Type":"ContainerStarted","Data":"f58f09072457d16e26c608d0f745bfd1518abefaacfa4aef5c1b1bacc82075ea"}
Sep 10 12:45:43.581683 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:43.581585    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:43.599959 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:43.599917    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth" podStartSLOduration=2.599905018 podStartE2EDuration="2.599905018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:43.598927126 +0000 UTC m=+918.121096182" watchObservedRunningTime="2026-09-10 12:45:43.599905018 +0000 UTC m=+918.122074108"
Sep 10 12:45:43.617730 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:43.617685    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7" podStartSLOduration=2.617671583 podStartE2EDuration="2.617671583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:43.616949959 +0000 UTC m=+918.139119010" watchObservedRunningTime="2026-09-10 12:45:43.617671583 +0000 UTC m=+918.139840680"
Sep 10 12:45:50.994682 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:50.994647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:45:54.588523 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:54.588491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:45:54.589047 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:45:54.588570    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:46:01.965374 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:01.965335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:02.370155 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:02.370115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:10.203805 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:10.203770    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:10.912213 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:10.912176    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:10.965282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:10.965247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"]
Sep 10 12:46:10.965574 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:10.965527    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv" podUID="b8d27f86-9097-4b0b-90ca-92ead9f5c537" containerName="payload-processing" containerID="cri-o://60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018" gracePeriod=30
Sep 10 12:46:11.002407 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.002375    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"]
Sep 10 12:46:11.002658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.002606    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz" podUID="f9746030-7804-466a-b5dd-e15ade63c7ce" containerName="payload-pre-processing" containerID="cri-o://75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c" gracePeriod=30
Sep 10 12:46:11.021858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.021819    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:11.128174 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.128136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:11.227473 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.227445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:11.255014 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.254989    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:46:11.289759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.289733    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:46:11.318319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.318287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:11.339465 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.339435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg45r\" (UniqueName: \"kubernetes.io/projected/b8d27f86-9097-4b0b-90ca-92ead9f5c537-kube-api-access-kg45r\") pod \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\" (UID: \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\") "
Sep 10 12:46:11.339587 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.339479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d27f86-9097-4b0b-90ca-92ead9f5c537-plugins-config-volume\") pod \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\" (UID: \"b8d27f86-9097-4b0b-90ca-92ead9f5c537\") "
Sep 10 12:46:11.340001 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.339975    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d27f86-9097-4b0b-90ca-92ead9f5c537-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8d27f86-9097-4b0b-90ca-92ead9f5c537" (UID: "b8d27f86-9097-4b0b-90ca-92ead9f5c537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:11.341706 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.341684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d27f86-9097-4b0b-90ca-92ead9f5c537-kube-api-access-kg45r" (OuterVolumeSpecName: "kube-api-access-kg45r") pod "b8d27f86-9097-4b0b-90ca-92ead9f5c537" (UID: "b8d27f86-9097-4b0b-90ca-92ead9f5c537"). InnerVolumeSpecName "kube-api-access-kg45r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:11.440987 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.440914    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrn9c\" (UniqueName: \"kubernetes.io/projected/f9746030-7804-466a-b5dd-e15ade63c7ce-kube-api-access-hrn9c\") pod \"f9746030-7804-466a-b5dd-e15ade63c7ce\" (UID: \"f9746030-7804-466a-b5dd-e15ade63c7ce\") "
Sep 10 12:46:11.441106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.440997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9746030-7804-466a-b5dd-e15ade63c7ce-plugins-config-volume\") pod \"f9746030-7804-466a-b5dd-e15ade63c7ce\" (UID: \"f9746030-7804-466a-b5dd-e15ade63c7ce\") "
Sep 10 12:46:11.441340 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.441322    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg45r\" (UniqueName: \"kubernetes.io/projected/b8d27f86-9097-4b0b-90ca-92ead9f5c537-kube-api-access-kg45r\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:11.441383 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.441348    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d27f86-9097-4b0b-90ca-92ead9f5c537-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:11.441383 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.441322    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9746030-7804-466a-b5dd-e15ade63c7ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9746030-7804-466a-b5dd-e15ade63c7ce" (UID: "f9746030-7804-466a-b5dd-e15ade63c7ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:11.443062 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.443043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9746030-7804-466a-b5dd-e15ade63c7ce-kube-api-access-hrn9c" (OuterVolumeSpecName: "kube-api-access-hrn9c") pod "f9746030-7804-466a-b5dd-e15ade63c7ce" (UID: "f9746030-7804-466a-b5dd-e15ade63c7ce"). InnerVolumeSpecName "kube-api-access-hrn9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:11.512954 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.512884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:11.542129 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.542098    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9746030-7804-466a-b5dd-e15ade63c7ce-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:11.542129 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.542124    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrn9c\" (UniqueName: \"kubernetes.io/projected/f9746030-7804-466a-b5dd-e15ade63c7ce-kube-api-access-hrn9c\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:11.667362 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.667317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:11.731932 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.731859    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8d27f86-9097-4b0b-90ca-92ead9f5c537" containerID="60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018" exitCode=0
Sep 10 12:46:11.731932 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.731902    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"
Sep 10 12:46:11.732156 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.731940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv" event={"ID":"b8d27f86-9097-4b0b-90ca-92ead9f5c537","Type":"ContainerDied","Data":"60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018"}
Sep 10 12:46:11.732156 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.731986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv" event={"ID":"b8d27f86-9097-4b0b-90ca-92ead9f5c537","Type":"ContainerDied","Data":"6704826d3ddda1ac775fdd67f4f285d21a27c051f646c68aa8c0e86be3471750"}
Sep 10 12:46:11.732156 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.732016    2577 scope.go:117] "RemoveContainer" containerID="60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018"
Sep 10 12:46:11.733389 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.733368    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9746030-7804-466a-b5dd-e15ade63c7ce" containerID="75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c" exitCode=0
Sep 10 12:46:11.733492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.733404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz" event={"ID":"f9746030-7804-466a-b5dd-e15ade63c7ce","Type":"ContainerDied","Data":"75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c"}
Sep 10 12:46:11.733492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.733407    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"
Sep 10 12:46:11.733492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.733436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz" event={"ID":"f9746030-7804-466a-b5dd-e15ade63c7ce","Type":"ContainerDied","Data":"03df7f802b335adc5ddf6be471e49c32735d4b7498800f053ba7af8bd0143bba"}
Sep 10 12:46:11.741763 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.741728    2577 scope.go:117] "RemoveContainer" containerID="60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018"
Sep 10 12:46:11.742082 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:46:11.742054    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018\": container with ID starting with 60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018 not found: ID does not exist" containerID="60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018"
Sep 10 12:46:11.742165 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.742092    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018"} err="failed to get container status \"60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018\": rpc error: code = NotFound desc = could not find container \"60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018\": container with ID starting with 60821cd927d65c68c11848a16209dcc8ff6a38c261037a58152d4b937d684018 not found: ID does not exist"
Sep 10 12:46:11.742165 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.742114    2577 scope.go:117] "RemoveContainer" containerID="75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c"
Sep 10 12:46:11.750873 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.750855    2577 scope.go:117] "RemoveContainer" containerID="75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c"
Sep 10 12:46:11.751151 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:46:11.751130    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c\": container with ID starting with 75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c not found: ID does not exist" containerID="75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c"
Sep 10 12:46:11.751221 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.751162    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c"} err="failed to get container status \"75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c\": rpc error: code = NotFound desc = could not find container \"75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c\": container with ID starting with 75393721d1459c8d667d30d3a552a966d191d0d055c54eb85ed695b2d6d5164c not found: ID does not exist"
Sep 10 12:46:11.778519 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.778494    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"]
Sep 10 12:46:11.788300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.788276    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5abff0-68f55b5545-9xbfv"]
Sep 10 12:46:11.816400 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.816368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"]
Sep 10 12:46:11.829468 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:11.829437    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5abff0-655db4b45d-8zzqz"]
Sep 10 12:46:12.000896 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.000866    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d27f86-9097-4b0b-90ca-92ead9f5c537" path="/var/lib/kubelet/pods/b8d27f86-9097-4b0b-90ca-92ead9f5c537/volumes"
Sep 10 12:46:12.001257 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.001244    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9746030-7804-466a-b5dd-e15ade63c7ce" path="/var/lib/kubelet/pods/f9746030-7804-466a-b5dd-e15ade63c7ce/volumes"
Sep 10 12:46:12.448711 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.448677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:12.451980 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.451943    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"]
Sep 10 12:46:12.452205 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.452182    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7" podUID="f0b2eb03-772f-42d7-8027-f0b4d08c3b7c" containerName="payload-pre-processing" containerID="cri-o://9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359" gracePeriod=30
Sep 10 12:46:12.455188 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.455163    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"]
Sep 10 12:46:12.455408 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.455379    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth" podUID="67a7cd23-e87b-477a-a068-fd8a32eeec0b" containerName="payload-processing" containerID="cri-o://ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d" gracePeriod=30
Sep 10 12:46:12.549106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.549076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:12.672486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.672453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:12.731201 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.731174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:12.735576 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.735554    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:46:12.738686 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.738660    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:46:12.741701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.741675    2577 generic.go:356] "Generic (PLEG): container finished" podID="67a7cd23-e87b-477a-a068-fd8a32eeec0b" containerID="ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d" exitCode=0
Sep 10 12:46:12.741793 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.741711    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"
Sep 10 12:46:12.741793 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.741749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth" event={"ID":"67a7cd23-e87b-477a-a068-fd8a32eeec0b","Type":"ContainerDied","Data":"ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d"}
Sep 10 12:46:12.741916 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.741795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth" event={"ID":"67a7cd23-e87b-477a-a068-fd8a32eeec0b","Type":"ContainerDied","Data":"2a94899c5de95e501509a8ce302b9db33a97c409070a3222c5f033f474296140"}
Sep 10 12:46:12.741916 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.741810    2577 scope.go:117] "RemoveContainer" containerID="ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d"
Sep 10 12:46:12.743147 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.743126    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0b2eb03-772f-42d7-8027-f0b4d08c3b7c" containerID="9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359" exitCode=0
Sep 10 12:46:12.743242 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.743165    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"
Sep 10 12:46:12.743242 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.743186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7" event={"ID":"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c","Type":"ContainerDied","Data":"9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359"}
Sep 10 12:46:12.743242 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.743216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7" event={"ID":"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c","Type":"ContainerDied","Data":"f58f09072457d16e26c608d0f745bfd1518abefaacfa4aef5c1b1bacc82075ea"}
Sep 10 12:46:12.750278 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.750259    2577 scope.go:117] "RemoveContainer" containerID="ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d"
Sep 10 12:46:12.750537 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:46:12.750519    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d\": container with ID starting with ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d not found: ID does not exist" containerID="ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d"
Sep 10 12:46:12.750591 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.750546    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d"} err="failed to get container status \"ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d\": rpc error: code = NotFound desc = could not find container \"ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d\": container with ID starting with ac68ed3302547ce0f03b208f7fc58e9669b7781ca405310883f1aaa66e60472d not found: ID does not exist"
Sep 10 12:46:12.750591 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.750562    2577 scope.go:117] "RemoveContainer" containerID="9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359"
Sep 10 12:46:12.758588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.758566    2577 scope.go:117] "RemoveContainer" containerID="9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359"
Sep 10 12:46:12.758878 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:46:12.758859    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359\": container with ID starting with 9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359 not found: ID does not exist" containerID="9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359"
Sep 10 12:46:12.758930 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.758883    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359"} err="failed to get container status \"9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359\": rpc error: code = NotFound desc = could not find container \"9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359\": container with ID starting with 9fb4f88da99e54cb8db0f127a563d5814b63d1fd8e71c57af92e7eb89b0e7359 not found: ID does not exist"
Sep 10 12:46:12.808108 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808078    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"]
Sep 10 12:46:12.808504 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808491    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b2eb03-772f-42d7-8027-f0b4d08c3b7c" containerName="payload-pre-processing"
Sep 10 12:46:12.808573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808506    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b2eb03-772f-42d7-8027-f0b4d08c3b7c" containerName="payload-pre-processing"
Sep 10 12:46:12.808573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808523    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a7cd23-e87b-477a-a068-fd8a32eeec0b" containerName="payload-processing"
Sep 10 12:46:12.808573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808529    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a7cd23-e87b-477a-a068-fd8a32eeec0b" containerName="payload-processing"
Sep 10 12:46:12.808573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808547    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9746030-7804-466a-b5dd-e15ade63c7ce" containerName="payload-pre-processing"
Sep 10 12:46:12.808573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808553    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9746030-7804-466a-b5dd-e15ade63c7ce" containerName="payload-pre-processing"
Sep 10 12:46:12.808573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808562    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d27f86-9097-4b0b-90ca-92ead9f5c537" containerName="payload-processing"
Sep 10 12:46:12.808573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808567    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d27f86-9097-4b0b-90ca-92ead9f5c537" containerName="payload-processing"
Sep 10 12:46:12.808828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808661    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9746030-7804-466a-b5dd-e15ade63c7ce" containerName="payload-pre-processing"
Sep 10 12:46:12.808828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808672    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d27f86-9097-4b0b-90ca-92ead9f5c537" containerName="payload-processing"
Sep 10 12:46:12.808828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808681    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b2eb03-772f-42d7-8027-f0b4d08c3b7c" containerName="payload-pre-processing"
Sep 10 12:46:12.808828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.808689    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a7cd23-e87b-477a-a068-fd8a32eeec0b" containerName="payload-processing"
Sep 10 12:46:12.813561 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.813536    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:12.816216 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.816170    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f85889\"/\"kube-root-ca.crt\""
Sep 10 12:46:12.816456 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.816438    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f85889\"/\"test-model-f85889-kserve-self-signed-certs\""
Sep 10 12:46:12.816578 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.816497    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f85889\"/\"default-dockercfg-kk9z7\""
Sep 10 12:46:12.816832 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.816814    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f85889\"/\"openshift-service-ca.crt\""
Sep 10 12:46:12.825643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.825620    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"]
Sep 10 12:46:12.856846 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.856816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv69d\" (UniqueName: \"kubernetes.io/projected/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-kube-api-access-mv69d\") pod \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\" (UID: \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\") "
Sep 10 12:46:12.856989 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.856849    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mv59\" (UniqueName: \"kubernetes.io/projected/67a7cd23-e87b-477a-a068-fd8a32eeec0b-kube-api-access-7mv59\") pod \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\" (UID: \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\") "
Sep 10 12:46:12.857077 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.857053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-plugins-config-volume\") pod \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\" (UID: \"f0b2eb03-772f-42d7-8027-f0b4d08c3b7c\") "
Sep 10 12:46:12.857147 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.857097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a7cd23-e87b-477a-a068-fd8a32eeec0b-plugins-config-volume\") pod \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\" (UID: \"67a7cd23-e87b-477a-a068-fd8a32eeec0b\") "
Sep 10 12:46:12.857465 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.857440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0b2eb03-772f-42d7-8027-f0b4d08c3b7c" (UID: "f0b2eb03-772f-42d7-8027-f0b4d08c3b7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:12.857564 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.857440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a7cd23-e87b-477a-a068-fd8a32eeec0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67a7cd23-e87b-477a-a068-fd8a32eeec0b" (UID: "67a7cd23-e87b-477a-a068-fd8a32eeec0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:12.859015 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.858992    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a7cd23-e87b-477a-a068-fd8a32eeec0b-kube-api-access-7mv59" (OuterVolumeSpecName: "kube-api-access-7mv59") pod "67a7cd23-e87b-477a-a068-fd8a32eeec0b" (UID: "67a7cd23-e87b-477a-a068-fd8a32eeec0b"). InnerVolumeSpecName "kube-api-access-7mv59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:12.859095 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.859021    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-kube-api-access-mv69d" (OuterVolumeSpecName: "kube-api-access-mv69d") pod "f0b2eb03-772f-42d7-8027-f0b4d08c3b7c" (UID: "f0b2eb03-772f-42d7-8027-f0b4d08c3b7c"). InnerVolumeSpecName "kube-api-access-mv69d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:12.958246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-home\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:12.958376 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-model-cache\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:12.958376 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwt2\" (UniqueName: \"kubernetes.io/projected/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-kube-api-access-vvwt2\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:12.958376 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tmp-dir\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:12.958498 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-dshm\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:12.958535 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:12.958581 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958570    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv69d\" (UniqueName: \"kubernetes.io/projected/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-kube-api-access-mv69d\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:12.958641 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958584    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mv59\" (UniqueName: \"kubernetes.io/projected/67a7cd23-e87b-477a-a068-fd8a32eeec0b-kube-api-access-7mv59\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:12.958641 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958616    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:12.958641 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:12.958626    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a7cd23-e87b-477a-a068-fd8a32eeec0b-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:13.059408 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.059382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-model-cache\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.059514 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.059417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwt2\" (UniqueName: \"kubernetes.io/projected/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-kube-api-access-vvwt2\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.059514 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.059439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tmp-dir\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.059514 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.059464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-dshm\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.059707 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.059555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.059707 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.059621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-home\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.059872 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.059848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tmp-dir\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.059872 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.059865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-model-cache\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.060104 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.060083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-home\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.062087 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.062065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-dshm\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.062364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.062340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.071721 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.071695    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"]
Sep 10 12:46:13.074405 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.074361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwt2\" (UniqueName: \"kubernetes.io/projected/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-kube-api-access-vvwt2\") pod \"test-model-f85889-kserve-d599c56f-gb9rm\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") " pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.076640 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.076618    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d7fcd5f6-59d67f8946-gv6g7"]
Sep 10 12:46:13.089120 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.089097    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"]
Sep 10 12:46:13.092912 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.092881    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7fcd5f6-5f7df5545c-2dfth"]
Sep 10 12:46:13.104159 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.104139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:13.125161 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.125135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:13.416288 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.416259    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"]
Sep 10 12:46:13.417456 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:46:13.417426    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod188a4e7c_bcb0_408e_af38_0f9f5cdbc478.slice/crio-7457db6555a0b38d14633ea6d4c2eaf17d58743eb3bc326708e4f3dc627bf822 WatchSource:0}: Error finding container 7457db6555a0b38d14633ea6d4c2eaf17d58743eb3bc326708e4f3dc627bf822: Status 404 returned error can't find the container with id 7457db6555a0b38d14633ea6d4c2eaf17d58743eb3bc326708e4f3dc627bf822
Sep 10 12:46:13.757533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.757438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm" event={"ID":"188a4e7c-bcb0-408e-af38-0f9f5cdbc478","Type":"ContainerStarted","Data":"07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb"}
Sep 10 12:46:13.757533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:13.757484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm" event={"ID":"188a4e7c-bcb0-408e-af38-0f9f5cdbc478","Type":"ContainerStarted","Data":"7457db6555a0b38d14633ea6d4c2eaf17d58743eb3bc326708e4f3dc627bf822"}
Sep 10 12:46:14.001009 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:14.000974    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a7cd23-e87b-477a-a068-fd8a32eeec0b" path="/var/lib/kubelet/pods/67a7cd23-e87b-477a-a068-fd8a32eeec0b/volumes"
Sep 10 12:46:14.001361 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:14.001347    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b2eb03-772f-42d7-8027-f0b4d08c3b7c" path="/var/lib/kubelet/pods/f0b2eb03-772f-42d7-8027-f0b4d08c3b7c/volumes"
Sep 10 12:46:14.360755 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:14.360721    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:14.580785 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:14.580753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:15.072709 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:15.072677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:16.736707 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:16.736671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:17.299316 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:17.299280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:17.410323 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:17.410273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:18.548584 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:18.548536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:18.649957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:18.649907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:18.974638 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:18.974522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:19.179746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.179712    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:19.275416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.275381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:19.294742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.294689    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm" podStartSLOduration=7.294672156 podStartE2EDuration="7.294672156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:46:13.784363414 +0000 UTC m=+948.306532466" watchObservedRunningTime="2026-09-10 12:46:19.294672156 +0000 UTC m=+953.816841207"
Sep 10 12:46:19.296512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.296491    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"]
Sep 10 12:46:19.301731 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.301711    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:19.304981 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.304952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3eca8974\""
Sep 10 12:46:19.314573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.314543    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"]
Sep 10 12:46:19.362396 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.362360    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"]
Sep 10 12:46:19.366098 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.366074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:19.368787 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.368764    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3eca8974-dockercfg-rgvhq\""
Sep 10 12:46:19.381253 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.381228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"]
Sep 10 12:46:19.417790 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.417757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxh6l\" (UniqueName: \"kubernetes.io/projected/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-kube-api-access-gxh6l\") pod \"payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq\" (UID: \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\") " pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:19.417952 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.417872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-plugins-config-volume\") pod \"payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq\" (UID: \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\") " pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:19.518999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.518962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxh6l\" (UniqueName: \"kubernetes.io/projected/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-kube-api-access-gxh6l\") pod \"payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq\" (UID: \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\") " pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:19.519166 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.519025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babd0c1a-1feb-4de8-8708-9b494dbd0111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5\" (UID: \"babd0c1a-1feb-4de8-8708-9b494dbd0111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:19.519166 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.519065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-plugins-config-volume\") pod \"payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq\" (UID: \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\") " pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:19.519166 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.519094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsmd\" (UniqueName: \"kubernetes.io/projected/babd0c1a-1feb-4de8-8708-9b494dbd0111-kube-api-access-8lsmd\") pod \"payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5\" (UID: \"babd0c1a-1feb-4de8-8708-9b494dbd0111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:19.519791 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.519770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-plugins-config-volume\") pod \"payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq\" (UID: \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\") " pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:19.527707 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.527652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxh6l\" (UniqueName: \"kubernetes.io/projected/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-kube-api-access-gxh6l\") pod \"payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq\" (UID: \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\") " pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:19.613159 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.613124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:19.620679 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.620648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babd0c1a-1feb-4de8-8708-9b494dbd0111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5\" (UID: \"babd0c1a-1feb-4de8-8708-9b494dbd0111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:19.620803 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.620766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsmd\" (UniqueName: \"kubernetes.io/projected/babd0c1a-1feb-4de8-8708-9b494dbd0111-kube-api-access-8lsmd\") pod \"payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5\" (UID: \"babd0c1a-1feb-4de8-8708-9b494dbd0111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:19.621347 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.621329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babd0c1a-1feb-4de8-8708-9b494dbd0111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5\" (UID: \"babd0c1a-1feb-4de8-8708-9b494dbd0111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:19.630803 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.630768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsmd\" (UniqueName: \"kubernetes.io/projected/babd0c1a-1feb-4de8-8708-9b494dbd0111-kube-api-access-8lsmd\") pod \"payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5\" (UID: \"babd0c1a-1feb-4de8-8708-9b494dbd0111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:19.676999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.676965    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:19.893543 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.893519    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"]
Sep 10 12:46:19.895769 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:46:19.895743    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ada163_4b71_4bf2_8f70_6b66882a2fd1.slice/crio-f27e4acfa78f340403b1fab489a3d85d102511655f2835e0285c1eeb8552ec85 WatchSource:0}: Error finding container f27e4acfa78f340403b1fab489a3d85d102511655f2835e0285c1eeb8552ec85: Status 404 returned error can't find the container with id f27e4acfa78f340403b1fab489a3d85d102511655f2835e0285c1eeb8552ec85
Sep 10 12:46:19.984427 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:19.984397    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"]
Sep 10 12:46:19.986427 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:46:19.986399    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabd0c1a_1feb_4de8_8708_9b494dbd0111.slice/crio-357f7ab07b9032d92bd978cd9d56655978b474dc56f5bcd43f48563e7b77cde4 WatchSource:0}: Error finding container 357f7ab07b9032d92bd978cd9d56655978b474dc56f5bcd43f48563e7b77cde4: Status 404 returned error can't find the container with id 357f7ab07b9032d92bd978cd9d56655978b474dc56f5bcd43f48563e7b77cde4
Sep 10 12:46:20.806467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:20.806404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5" event={"ID":"babd0c1a-1feb-4de8-8708-9b494dbd0111","Type":"ContainerStarted","Data":"7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7"}
Sep 10 12:46:20.806467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:20.806467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5" event={"ID":"babd0c1a-1feb-4de8-8708-9b494dbd0111","Type":"ContainerStarted","Data":"357f7ab07b9032d92bd978cd9d56655978b474dc56f5bcd43f48563e7b77cde4"}
Sep 10 12:46:20.807025 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:20.806525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:20.808353 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:20.808324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq" event={"ID":"d3ada163-4b71-4bf2-8f70-6b66882a2fd1","Type":"ContainerStarted","Data":"feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860"}
Sep 10 12:46:20.808507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:20.808374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq" event={"ID":"d3ada163-4b71-4bf2-8f70-6b66882a2fd1","Type":"ContainerStarted","Data":"f27e4acfa78f340403b1fab489a3d85d102511655f2835e0285c1eeb8552ec85"}
Sep 10 12:46:20.808507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:20.808416    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:20.840223 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:20.840157    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5" podStartSLOduration=1.8401346570000001 podStartE2EDuration="1.840134657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:46:20.836888529 +0000 UTC m=+955.359057593" watchObservedRunningTime="2026-09-10 12:46:20.840134657 +0000 UTC m=+955.362303710"
Sep 10 12:46:20.870316 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:20.870257    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq" podStartSLOduration=1.870234991 podStartE2EDuration="1.870234991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:46:20.869038527 +0000 UTC m=+955.391207579" watchObservedRunningTime="2026-09-10 12:46:20.870234991 +0000 UTC m=+955.392404044"
Sep 10 12:46:23.125316 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:23.125279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:23.125796 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:23.125337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:23.143134 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:23.143095    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:23.836423 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:23.836392    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:46:30.420911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:30.420869    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:31.815865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:31.815823    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:31.816318 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:31.816275    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:47.184388 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:47.184350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:49.901274 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:49.901197    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:50.016764 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.016729    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"]
Sep 10 12:46:50.017019 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.016984    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq" podUID="d3ada163-4b71-4bf2-8f70-6b66882a2fd1" containerName="payload-processing" containerID="cri-o://feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860" gracePeriod=30
Sep 10 12:46:50.045002 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.044979    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"]
Sep 10 12:46:50.045180 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.045162    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5" podUID="babd0c1a-1feb-4de8-8708-9b494dbd0111" containerName="payload-pre-processing" containerID="cri-o://7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7" gracePeriod=30
Sep 10 12:46:50.108789 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.108747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:50.221742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.221707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:50.312797 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.312764    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:50.329378 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.329356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:50.336531 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.336514    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:50.395030 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.395007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxh6l\" (UniqueName: \"kubernetes.io/projected/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-kube-api-access-gxh6l\") pod \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\" (UID: \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\") "
Sep 10 12:46:50.395190 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.395037    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-plugins-config-volume\") pod \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\" (UID: \"d3ada163-4b71-4bf2-8f70-6b66882a2fd1\") "
Sep 10 12:46:50.395190 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.395077    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lsmd\" (UniqueName: \"kubernetes.io/projected/babd0c1a-1feb-4de8-8708-9b494dbd0111-kube-api-access-8lsmd\") pod \"babd0c1a-1feb-4de8-8708-9b494dbd0111\" (UID: \"babd0c1a-1feb-4de8-8708-9b494dbd0111\") "
Sep 10 12:46:50.395190 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.395150    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babd0c1a-1feb-4de8-8708-9b494dbd0111-plugins-config-volume\") pod \"babd0c1a-1feb-4de8-8708-9b494dbd0111\" (UID: \"babd0c1a-1feb-4de8-8708-9b494dbd0111\") "
Sep 10 12:46:50.395527 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.395502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3ada163-4b71-4bf2-8f70-6b66882a2fd1" (UID: "d3ada163-4b71-4bf2-8f70-6b66882a2fd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:50.395925 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.395581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babd0c1a-1feb-4de8-8708-9b494dbd0111-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "babd0c1a-1feb-4de8-8708-9b494dbd0111" (UID: "babd0c1a-1feb-4de8-8708-9b494dbd0111"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:50.397227 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.397204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-kube-api-access-gxh6l" (OuterVolumeSpecName: "kube-api-access-gxh6l") pod "d3ada163-4b71-4bf2-8f70-6b66882a2fd1" (UID: "d3ada163-4b71-4bf2-8f70-6b66882a2fd1"). InnerVolumeSpecName "kube-api-access-gxh6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:50.397315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.397234    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babd0c1a-1feb-4de8-8708-9b494dbd0111-kube-api-access-8lsmd" (OuterVolumeSpecName: "kube-api-access-8lsmd") pod "babd0c1a-1feb-4de8-8708-9b494dbd0111" (UID: "babd0c1a-1feb-4de8-8708-9b494dbd0111"). InnerVolumeSpecName "kube-api-access-8lsmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:50.407273 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.407219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:50.496120 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.496082    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:50.496120 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.496110    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsmd\" (UniqueName: \"kubernetes.io/projected/babd0c1a-1feb-4de8-8708-9b494dbd0111-kube-api-access-8lsmd\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:50.496120 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.496120    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babd0c1a-1feb-4de8-8708-9b494dbd0111-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:50.496423 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.496133    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxh6l\" (UniqueName: \"kubernetes.io/projected/d3ada163-4b71-4bf2-8f70-6b66882a2fd1-kube-api-access-gxh6l\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:46:50.547476 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.547441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:50.975270 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.975238    2577 generic.go:356] "Generic (PLEG): container finished" podID="babd0c1a-1feb-4de8-8708-9b494dbd0111" containerID="7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7" exitCode=0
Sep 10 12:46:50.975733 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.975285    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"
Sep 10 12:46:50.975733 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.975306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5" event={"ID":"babd0c1a-1feb-4de8-8708-9b494dbd0111","Type":"ContainerDied","Data":"7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7"}
Sep 10 12:46:50.975733 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.975352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5" event={"ID":"babd0c1a-1feb-4de8-8708-9b494dbd0111","Type":"ContainerDied","Data":"357f7ab07b9032d92bd978cd9d56655978b474dc56f5bcd43f48563e7b77cde4"}
Sep 10 12:46:50.975733 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.975375    2577 scope.go:117] "RemoveContainer" containerID="7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7"
Sep 10 12:46:50.976763 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.976742    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3ada163-4b71-4bf2-8f70-6b66882a2fd1" containerID="feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860" exitCode=0
Sep 10 12:46:50.976855 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.976786    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"
Sep 10 12:46:50.976855 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.976828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq" event={"ID":"d3ada163-4b71-4bf2-8f70-6b66882a2fd1","Type":"ContainerDied","Data":"feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860"}
Sep 10 12:46:50.976934 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.976896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq" event={"ID":"d3ada163-4b71-4bf2-8f70-6b66882a2fd1","Type":"ContainerDied","Data":"f27e4acfa78f340403b1fab489a3d85d102511655f2835e0285c1eeb8552ec85"}
Sep 10 12:46:50.986133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.986120    2577 scope.go:117] "RemoveContainer" containerID="7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7"
Sep 10 12:46:50.986409 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:46:50.986389    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7\": container with ID starting with 7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7 not found: ID does not exist" containerID="7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7"
Sep 10 12:46:50.986473 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.986417    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7"} err="failed to get container status \"7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7\": rpc error: code = NotFound desc = could not find container \"7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7\": container with ID starting with 7f94565bf235e99676a2f7edce93706c2b37b218fee2a669163a3a8924084cd7 not found: ID does not exist"
Sep 10 12:46:50.986473 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.986434    2577 scope.go:117] "RemoveContainer" containerID="feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860"
Sep 10 12:46:50.994476 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.994460    2577 scope.go:117] "RemoveContainer" containerID="feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860"
Sep 10 12:46:50.994793 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:46:50.994774    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860\": container with ID starting with feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860 not found: ID does not exist" containerID="feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860"
Sep 10 12:46:50.994852 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.994799    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860"} err="failed to get container status \"feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860\": rpc error: code = NotFound desc = could not find container \"feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860\": container with ID starting with feab999601b5944b19bf20f371bdafc8c4e648c70520aab3149360c678d74860 not found: ID does not exist"
Sep 10 12:46:50.999544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:50.999523    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"]
Sep 10 12:46:51.004233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:51.004211    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3eca8974-767778995d-xq6m5"]
Sep 10 12:46:51.015698 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:51.015678    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"]
Sep 10 12:46:51.020093 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:51.020075    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3eca8974-7d9767cb65-fwfjq"]
Sep 10 12:46:51.227309 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:51.227230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:51.321889 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:51.321857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:51.457241 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:51.457210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:52.000206 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:52.000175    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babd0c1a-1feb-4de8-8708-9b494dbd0111" path="/var/lib/kubelet/pods/babd0c1a-1feb-4de8-8708-9b494dbd0111/volumes"
Sep 10 12:46:52.000569 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:52.000533    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ada163-4b71-4bf2-8f70-6b66882a2fd1" path="/var/lib/kubelet/pods/d3ada163-4b71-4bf2-8f70-6b66882a2fd1/volumes"
Sep 10 12:46:53.033444 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:53.033404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:57.026149 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:57.026116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:57.341157 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:57.341066    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:57.827588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:57.827552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:58.593643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:58.593563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:46:58.954728 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:46:58.954628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:02.372651 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:02.372543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:02.526317 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:02.526272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:03.658455 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:03.658418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:03.769300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:03.769263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:03.905340 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:03.905310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:04.346845 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.346809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:04.585715 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.585676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:04.645660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.645562    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"]
Sep 10 12:47:04.645998 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.645984    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babd0c1a-1feb-4de8-8708-9b494dbd0111" containerName="payload-pre-processing"
Sep 10 12:47:04.646045 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.645999    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="babd0c1a-1feb-4de8-8708-9b494dbd0111" containerName="payload-pre-processing"
Sep 10 12:47:04.646045 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.646009    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ada163-4b71-4bf2-8f70-6b66882a2fd1" containerName="payload-processing"
Sep 10 12:47:04.646045 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.646014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ada163-4b71-4bf2-8f70-6b66882a2fd1" containerName="payload-processing"
Sep 10 12:47:04.646143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.646103    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="babd0c1a-1feb-4de8-8708-9b494dbd0111" containerName="payload-pre-processing"
Sep 10 12:47:04.646143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.646112    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ada163-4b71-4bf2-8f70-6b66882a2fd1" containerName="payload-processing"
Sep 10 12:47:04.649377 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.649362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:04.656852 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.656831    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2d4635-dockercfg-8snnw\""
Sep 10 12:47:04.656977 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.656968    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2d4635\""
Sep 10 12:47:04.676980 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.676954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"]
Sep 10 12:47:04.732348 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.732307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:04.832994 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.832963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp\" (UID: \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:04.833161 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.833007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9489h\" (UniqueName: \"kubernetes.io/projected/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-kube-api-access-9489h\") pod \"payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp\" (UID: \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:04.934032 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.933954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9489h\" (UniqueName: \"kubernetes.io/projected/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-kube-api-access-9489h\") pod \"payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp\" (UID: \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:04.934178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.934072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp\" (UID: \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:04.934721 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.934703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp\" (UID: \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:04.948985 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.948962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9489h\" (UniqueName: \"kubernetes.io/projected/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-kube-api-access-9489h\") pod \"payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp\" (UID: \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\") " pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:04.960055 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:04.960032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:05.094532 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:05.094496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:05.303698 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:05.303670    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"]
Sep 10 12:47:05.305359 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:47:05.305332    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f63b4b4_78df_45ef_b2f5_7d6327f3fa37.slice/crio-24962db2d9419da8926426eadb05b2246b66fc9f9888825fcc83da63e53efd72 WatchSource:0}: Error finding container 24962db2d9419da8926426eadb05b2246b66fc9f9888825fcc83da63e53efd72: Status 404 returned error can't find the container with id 24962db2d9419da8926426eadb05b2246b66fc9f9888825fcc83da63e53efd72
Sep 10 12:47:06.061722 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:06.061689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp" event={"ID":"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37","Type":"ContainerStarted","Data":"6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470"}
Sep 10 12:47:06.062082 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:06.061740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp" event={"ID":"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37","Type":"ContainerStarted","Data":"24962db2d9419da8926426eadb05b2246b66fc9f9888825fcc83da63e53efd72"}
Sep 10 12:47:06.062082 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:06.061828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:06.095407 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:06.095362    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp" podStartSLOduration=2.09534434 podStartE2EDuration="2.09534434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:47:06.093369503 +0000 UTC m=+1000.615538572" watchObservedRunningTime="2026-09-10 12:47:06.09534434 +0000 UTC m=+1000.617513395"
Sep 10 12:47:10.108068 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:10.108031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:17.068835 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:17.068797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:47:17.454285 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:17.454195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:27.295499 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:27.295468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:28.146089 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:28.146047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:31.109042 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:31.109001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:38.741906 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:38.741871    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:38.868810 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:38.868775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:38.977467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:38.977427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:41.621767 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.621737    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"]
Sep 10 12:47:41.627083 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.627065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.645658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.645631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-714b74\"/\"openshift-service-ca.crt\""
Sep 10 12:47:41.645783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.645631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-714b74\"/\"kube-root-ca.crt\""
Sep 10 12:47:41.645783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.645630    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-714b74\"/\"test-model-714b74-kserve-self-signed-certs\""
Sep 10 12:47:41.645783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.645675    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-714b74\"/\"default-dockercfg-ccqvn\""
Sep 10 12:47:41.654878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.654858    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"]
Sep 10 12:47:41.717583 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.717556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:41.778702 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.778673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-tmp-dir\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.778893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.778715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-home\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.778893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.778735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-model-cache\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.778893 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.778827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kkh4\" (UniqueName: \"kubernetes.io/projected/90b1403c-dede-4b7d-aca2-8d617d82d610-kube-api-access-9kkh4\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.779076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.778901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-dshm\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.779076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.778938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b1403c-dede-4b7d-aca2-8d617d82d610-tls-certs\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.879978 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.879899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-dshm\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.879978 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.879953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b1403c-dede-4b7d-aca2-8d617d82d610-tls-certs\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.880189 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.880002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-tmp-dir\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.880189 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.880030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-home\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.880189 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.880055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-model-cache\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.880189 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.880117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kkh4\" (UniqueName: \"kubernetes.io/projected/90b1403c-dede-4b7d-aca2-8d617d82d610-kube-api-access-9kkh4\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.880392 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.880372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-home\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.880448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.880418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-tmp-dir\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.880499 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.880459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-model-cache\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.882262 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.882242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-dshm\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.882535 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.882514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b1403c-dede-4b7d-aca2-8d617d82d610-tls-certs\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.892684 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.892660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kkh4\" (UniqueName: \"kubernetes.io/projected/90b1403c-dede-4b7d-aca2-8d617d82d610-kube-api-access-9kkh4\") pod \"test-model-714b74-kserve-57f898f5ff-tgqs4\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") " pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:41.936666 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:41.936643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:42.214145 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:42.214122    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"]
Sep 10 12:47:42.215413 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:47:42.215387    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90b1403c_dede_4b7d_aca2_8d617d82d610.slice/crio-e48f15636a7603a34a6fd586868fc683572ca57eb4fa3e59be14b9b2c3122dcd WatchSource:0}: Error finding container e48f15636a7603a34a6fd586868fc683572ca57eb4fa3e59be14b9b2c3122dcd: Status 404 returned error can't find the container with id e48f15636a7603a34a6fd586868fc683572ca57eb4fa3e59be14b9b2c3122dcd
Sep 10 12:47:42.260634 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:42.260608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4" event={"ID":"90b1403c-dede-4b7d-aca2-8d617d82d610","Type":"ContainerStarted","Data":"e48f15636a7603a34a6fd586868fc683572ca57eb4fa3e59be14b9b2c3122dcd"}
Sep 10 12:47:43.274484 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:43.274447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4" event={"ID":"90b1403c-dede-4b7d-aca2-8d617d82d610","Type":"ContainerStarted","Data":"bc205e6e96383af0c830e13b5768423c67726978753cc3ae9160d0af09044bcf"}
Sep 10 12:47:43.297947 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:43.297881    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4" podStartSLOduration=2.297867068 podStartE2EDuration="2.297867068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:47:43.297202823 +0000 UTC m=+1037.819371878" watchObservedRunningTime="2026-09-10 12:47:43.297867068 +0000 UTC m=+1037.820036122"
Sep 10 12:47:50.304397 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:50.304363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:47:51.937684 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:51.937647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:51.938125 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:51.937737    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:51.950300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:51.950277    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:52.333501 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:52.333471    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:47:54.866704 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:47:54.866669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:02.075580 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:02.075552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:02.214762 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:02.214703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:02.397767 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:02.397626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:06.858979 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:06.858929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:12.904313 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:12.904275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:13.748904 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:13.748861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:22.208719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:22.208614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:22.369238 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:22.369206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:22.512184 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:22.512079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:22.591399 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:22.591367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:23.294656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:23.294582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:23.401327 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:23.401289    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:25.121541 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:25.121496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:30.265266 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:30.265227    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:32.155074 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:32.155035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:32.252404 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:32.252374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:32.855176 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:32.855138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:34.852443 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:34.852409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:34.920974 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:34.920938    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"]
Sep 10 12:48:34.921236 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:34.921205    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp" podUID="0f63b4b4-78df-45ef-b2f5-7d6327f3fa37" containerName="payload-processing" containerID="cri-o://6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470" gracePeriod=30
Sep 10 12:48:34.991699 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:34.991672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:35.096697 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.096613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:35.182142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.182118    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:48:35.182632 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.182578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-plugins-config-volume\") pod \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\" (UID: \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\") "
Sep 10 12:48:35.182699 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.182658    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9489h\" (UniqueName: \"kubernetes.io/projected/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-kube-api-access-9489h\") pod \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\" (UID: \"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37\") "
Sep 10 12:48:35.182982 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.182952    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f63b4b4-78df-45ef-b2f5-7d6327f3fa37" (UID: "0f63b4b4-78df-45ef-b2f5-7d6327f3fa37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:48:35.183139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.183126    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:48:35.184590 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.184565    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-kube-api-access-9489h" (OuterVolumeSpecName: "kube-api-access-9489h") pod "0f63b4b4-78df-45ef-b2f5-7d6327f3fa37" (UID: "0f63b4b4-78df-45ef-b2f5-7d6327f3fa37"). InnerVolumeSpecName "kube-api-access-9489h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:48:35.199690 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.199669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:35.284005 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.283970    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9489h\" (UniqueName: \"kubernetes.io/projected/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37-kube-api-access-9489h\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:48:35.300723 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.300693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:35.557499 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.557469    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f63b4b4-78df-45ef-b2f5-7d6327f3fa37" containerID="6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470" exitCode=0
Sep 10 12:48:35.557724 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.557516    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"
Sep 10 12:48:35.557724 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.557557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp" event={"ID":"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37","Type":"ContainerDied","Data":"6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470"}
Sep 10 12:48:35.557724 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.557642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp" event={"ID":"0f63b4b4-78df-45ef-b2f5-7d6327f3fa37","Type":"ContainerDied","Data":"24962db2d9419da8926426eadb05b2246b66fc9f9888825fcc83da63e53efd72"}
Sep 10 12:48:35.557724 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.557675    2577 scope.go:117] "RemoveContainer" containerID="6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470"
Sep 10 12:48:35.567087 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.567070    2577 scope.go:117] "RemoveContainer" containerID="6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470"
Sep 10 12:48:35.567436 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:48:35.567399    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470\": container with ID starting with 6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470 not found: ID does not exist" containerID="6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470"
Sep 10 12:48:35.567514 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.567444    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470"} err="failed to get container status \"6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470\": rpc error: code = NotFound desc = could not find container \"6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470\": container with ID starting with 6896a4cd801ca2f036b190092527addad4dd327ced4cb94f34e0528f537e0470 not found: ID does not exist"
Sep 10 12:48:35.594573 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.594544    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"]
Sep 10 12:48:35.597065 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.597046    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2d4635-57f968f597-5nlrp"]
Sep 10 12:48:35.678687 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:35.678636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:36.001285 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:36.001199    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f63b4b4-78df-45ef-b2f5-7d6327f3fa37" path="/var/lib/kubelet/pods/0f63b4b4-78df-45ef-b2f5-7d6327f3fa37/volumes"
Sep 10 12:48:40.510379 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:40.510343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:40.622130 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:40.622102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:40.907427 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:40.907384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:41.039900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:41.039872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:41.218114 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:41.218018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:42.013146 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:42.013117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:45.759832 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:45.759798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:48.273553 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:48.273507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:48.361381 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:48.361341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:48.617163 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:48.617081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:48.954384 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:48.954307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:49.043710 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:49.043677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:49.147041 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:49.147014    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:49.448455 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:49.448418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:49.715447 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:49.715367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:50.386448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:50.386402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:50.511121 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:50.511084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:50.871309 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:50.871278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:51.204032 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:51.203939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:51.365171 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:51.365139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:51.487391 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:51.487298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:51.732684 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:51.732652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:52.015188 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.015158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:48:52.032176 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.032146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"]
Sep 10 12:48:52.032578 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.032565    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f63b4b4-78df-45ef-b2f5-7d6327f3fa37" containerName="payload-processing"
Sep 10 12:48:52.032660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.032580    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f63b4b4-78df-45ef-b2f5-7d6327f3fa37" containerName="payload-processing"
Sep 10 12:48:52.032701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.032670    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f63b4b4-78df-45ef-b2f5-7d6327f3fa37" containerName="payload-processing"
Sep 10 12:48:52.037363 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.037342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:52.046365 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.046348    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-006b1977-dockercfg-kwk6p\""
Sep 10 12:48:52.060402 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.060380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-006b1977\""
Sep 10 12:48:52.081469 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.081449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"]
Sep 10 12:48:52.120389 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.120362    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"]
Sep 10 12:48:52.124740 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.124720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:52.137562 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.137537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c212216-7b90-4228-9bf1-cf2718514d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv\" (UID: \"7c212216-7b90-4228-9bf1-cf2718514d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:52.137668 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.137604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktlv\" (UniqueName: \"kubernetes.io/projected/7c212216-7b90-4228-9bf1-cf2718514d4b-kube-api-access-dktlv\") pod \"payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv\" (UID: \"7c212216-7b90-4228-9bf1-cf2718514d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:52.147673 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.147651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"]
Sep 10 12:48:52.239094 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.239058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9322bdc0-598b-471d-a024-07ce8d383891-plugins-config-volume\") pod \"payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5\" (UID: \"9322bdc0-598b-471d-a024-07ce8d383891\") " pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:52.239294 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.239125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4cxn\" (UniqueName: \"kubernetes.io/projected/9322bdc0-598b-471d-a024-07ce8d383891-kube-api-access-f4cxn\") pod \"payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5\" (UID: \"9322bdc0-598b-471d-a024-07ce8d383891\") " pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:52.239294 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.239150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c212216-7b90-4228-9bf1-cf2718514d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv\" (UID: \"7c212216-7b90-4228-9bf1-cf2718514d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:52.239294 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.239209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktlv\" (UniqueName: \"kubernetes.io/projected/7c212216-7b90-4228-9bf1-cf2718514d4b-kube-api-access-dktlv\") pod \"payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv\" (UID: \"7c212216-7b90-4228-9bf1-cf2718514d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:52.239860 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.239837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c212216-7b90-4228-9bf1-cf2718514d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv\" (UID: \"7c212216-7b90-4228-9bf1-cf2718514d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:52.259535 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.259506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktlv\" (UniqueName: \"kubernetes.io/projected/7c212216-7b90-4228-9bf1-cf2718514d4b-kube-api-access-dktlv\") pod \"payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv\" (UID: \"7c212216-7b90-4228-9bf1-cf2718514d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:52.340397 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.340368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9322bdc0-598b-471d-a024-07ce8d383891-plugins-config-volume\") pod \"payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5\" (UID: \"9322bdc0-598b-471d-a024-07ce8d383891\") " pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:52.340530 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.340410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4cxn\" (UniqueName: \"kubernetes.io/projected/9322bdc0-598b-471d-a024-07ce8d383891-kube-api-access-f4cxn\") pod \"payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5\" (UID: \"9322bdc0-598b-471d-a024-07ce8d383891\") " pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:52.341111 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.341089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9322bdc0-598b-471d-a024-07ce8d383891-plugins-config-volume\") pod \"payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5\" (UID: \"9322bdc0-598b-471d-a024-07ce8d383891\") " pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:52.347451 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.347432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:52.361030 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.361005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4cxn\" (UniqueName: \"kubernetes.io/projected/9322bdc0-598b-471d-a024-07ce8d383891-kube-api-access-f4cxn\") pod \"payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5\" (UID: \"9322bdc0-598b-471d-a024-07ce8d383891\") " pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:52.435444 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.435416    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:52.642384 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.642357    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"]
Sep 10 12:48:52.643125 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:48:52.643098    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c212216_7b90_4228_9bf1_cf2718514d4b.slice/crio-7375858a8b962b2d0701dd506d4def24d3fc9184cf492ba55a98558a7367267e WatchSource:0}: Error finding container 7375858a8b962b2d0701dd506d4def24d3fc9184cf492ba55a98558a7367267e: Status 404 returned error can't find the container with id 7375858a8b962b2d0701dd506d4def24d3fc9184cf492ba55a98558a7367267e
Sep 10 12:48:52.657074 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.657040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv" event={"ID":"7c212216-7b90-4228-9bf1-cf2718514d4b","Type":"ContainerStarted","Data":"7375858a8b962b2d0701dd506d4def24d3fc9184cf492ba55a98558a7367267e"}
Sep 10 12:48:52.723691 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:52.723663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"]
Sep 10 12:48:52.725461 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:48:52.725438    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9322bdc0_598b_471d_a024_07ce8d383891.slice/crio-871acd7dbddbbf5dc0805fb9894538d170943bf330bdde758cf4aaf86f41d9fb WatchSource:0}: Error finding container 871acd7dbddbbf5dc0805fb9894538d170943bf330bdde758cf4aaf86f41d9fb: Status 404 returned error can't find the container with id 871acd7dbddbbf5dc0805fb9894538d170943bf330bdde758cf4aaf86f41d9fb
Sep 10 12:48:53.666996 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:53.666957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv" event={"ID":"7c212216-7b90-4228-9bf1-cf2718514d4b","Type":"ContainerStarted","Data":"959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305"}
Sep 10 12:48:53.667503 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:53.667093    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:48:53.668804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:53.668781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5" event={"ID":"9322bdc0-598b-471d-a024-07ce8d383891","Type":"ContainerStarted","Data":"df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87"}
Sep 10 12:48:53.668926 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:53.668836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5" event={"ID":"9322bdc0-598b-471d-a024-07ce8d383891","Type":"ContainerStarted","Data":"871acd7dbddbbf5dc0805fb9894538d170943bf330bdde758cf4aaf86f41d9fb"}
Sep 10 12:48:53.668926 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:53.668875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:48:53.691339 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:53.691292    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv" podStartSLOduration=2.691279318 podStartE2EDuration="2.691279318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:48:53.688342286 +0000 UTC m=+1108.210511338" watchObservedRunningTime="2026-09-10 12:48:53.691279318 +0000 UTC m=+1108.213448369"
Sep 10 12:48:53.710733 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:48:53.710692    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5" podStartSLOduration=1.7106750769999999 podStartE2EDuration="1.710675077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:48:53.709122372 +0000 UTC m=+1108.231291425" watchObservedRunningTime="2026-09-10 12:48:53.710675077 +0000 UTC m=+1108.232844173"
Sep 10 12:49:01.850467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:01.850422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:04.677652 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:04.677619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:49:04.678128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:04.677708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:49:13.868822 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:13.868783    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:14.546820 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:14.546786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:14.631944 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:14.631909    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:14.974368 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:14.974285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:16.600283 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:16.600245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:22.469757 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:22.469720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:22.572015 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:22.571970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:27.825679 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:27.825637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:31.126872 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:31.126839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:32.474166 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:32.474133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:33.466219 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.466184    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:33.521674 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.521641    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"]
Sep 10 12:49:33.522092 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.521899    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5" podUID="9322bdc0-598b-471d-a024-07ce8d383891" containerName="payload-processing" containerID="cri-o://df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87" gracePeriod=30
Sep 10 12:49:33.530751 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.530719    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"]
Sep 10 12:49:33.530981 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.530954    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv" podUID="7c212216-7b90-4228-9bf1-cf2718514d4b" containerName="payload-pre-processing" containerID="cri-o://959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305" gracePeriod=30
Sep 10 12:49:33.684119 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.684084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:33.786255 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.786224    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:33.808074 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.808053    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:49:33.810914 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.810895    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:49:33.876132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.876094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:33.903353 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.903318    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c212216-7b90-4228-9bf1-cf2718514d4b" containerID="959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305" exitCode=0
Sep 10 12:49:33.903492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.903360    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"
Sep 10 12:49:33.903492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.903401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv" event={"ID":"7c212216-7b90-4228-9bf1-cf2718514d4b","Type":"ContainerDied","Data":"959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305"}
Sep 10 12:49:33.903492 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.903456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv" event={"ID":"7c212216-7b90-4228-9bf1-cf2718514d4b","Type":"ContainerDied","Data":"7375858a8b962b2d0701dd506d4def24d3fc9184cf492ba55a98558a7367267e"}
Sep 10 12:49:33.903670 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.903492    2577 scope.go:117] "RemoveContainer" containerID="959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305"
Sep 10 12:49:33.904873 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.904852    2577 generic.go:356] "Generic (PLEG): container finished" podID="9322bdc0-598b-471d-a024-07ce8d383891" containerID="df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87" exitCode=0
Sep 10 12:49:33.904955 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.904896    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"
Sep 10 12:49:33.904955 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.904931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5" event={"ID":"9322bdc0-598b-471d-a024-07ce8d383891","Type":"ContainerDied","Data":"df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87"}
Sep 10 12:49:33.905052 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.904973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5" event={"ID":"9322bdc0-598b-471d-a024-07ce8d383891","Type":"ContainerDied","Data":"871acd7dbddbbf5dc0805fb9894538d170943bf330bdde758cf4aaf86f41d9fb"}
Sep 10 12:49:33.912748 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.912728    2577 scope.go:117] "RemoveContainer" containerID="959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305"
Sep 10 12:49:33.913048 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:49:33.913020    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305\": container with ID starting with 959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305 not found: ID does not exist" containerID="959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305"
Sep 10 12:49:33.913124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.913053    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305"} err="failed to get container status \"959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305\": rpc error: code = NotFound desc = could not find container \"959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305\": container with ID starting with 959ce484dcb1833701e2f45761ee4fbdb9ff3f9e1948f2bda4c07dcc22613305 not found: ID does not exist"
Sep 10 12:49:33.913124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.913076    2577 scope.go:117] "RemoveContainer" containerID="df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87"
Sep 10 12:49:33.921337 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.921321    2577 scope.go:117] "RemoveContainer" containerID="df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87"
Sep 10 12:49:33.921619 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:49:33.921578    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87\": container with ID starting with df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87 not found: ID does not exist" containerID="df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87"
Sep 10 12:49:33.921689 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.921618    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87"} err="failed to get container status \"df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87\": rpc error: code = NotFound desc = could not find container \"df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87\": container with ID starting with df72d97909cf66d8855a5a603fe3b10c60f0e59a9ce616d86851fdf1bf6ebc87 not found: ID does not exist"
Sep 10 12:49:33.938349 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.938331    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4cxn\" (UniqueName: \"kubernetes.io/projected/9322bdc0-598b-471d-a024-07ce8d383891-kube-api-access-f4cxn\") pod \"9322bdc0-598b-471d-a024-07ce8d383891\" (UID: \"9322bdc0-598b-471d-a024-07ce8d383891\") "
Sep 10 12:49:33.938449 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.938359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c212216-7b90-4228-9bf1-cf2718514d4b-plugins-config-volume\") pod \"7c212216-7b90-4228-9bf1-cf2718514d4b\" (UID: \"7c212216-7b90-4228-9bf1-cf2718514d4b\") "
Sep 10 12:49:33.938449 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.938384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dktlv\" (UniqueName: \"kubernetes.io/projected/7c212216-7b90-4228-9bf1-cf2718514d4b-kube-api-access-dktlv\") pod \"7c212216-7b90-4228-9bf1-cf2718514d4b\" (UID: \"7c212216-7b90-4228-9bf1-cf2718514d4b\") "
Sep 10 12:49:33.938449 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.938404    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9322bdc0-598b-471d-a024-07ce8d383891-plugins-config-volume\") pod \"9322bdc0-598b-471d-a024-07ce8d383891\" (UID: \"9322bdc0-598b-471d-a024-07ce8d383891\") "
Sep 10 12:49:33.938769 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.938739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c212216-7b90-4228-9bf1-cf2718514d4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c212216-7b90-4228-9bf1-cf2718514d4b" (UID: "7c212216-7b90-4228-9bf1-cf2718514d4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:49:33.938833 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.938814    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9322bdc0-598b-471d-a024-07ce8d383891-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9322bdc0-598b-471d-a024-07ce8d383891" (UID: "9322bdc0-598b-471d-a024-07ce8d383891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:49:33.940415 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.940397    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c212216-7b90-4228-9bf1-cf2718514d4b-kube-api-access-dktlv" (OuterVolumeSpecName: "kube-api-access-dktlv") pod "7c212216-7b90-4228-9bf1-cf2718514d4b" (UID: "7c212216-7b90-4228-9bf1-cf2718514d4b"). InnerVolumeSpecName "kube-api-access-dktlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:49:33.940472 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:33.940451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9322bdc0-598b-471d-a024-07ce8d383891-kube-api-access-f4cxn" (OuterVolumeSpecName: "kube-api-access-f4cxn") pod "9322bdc0-598b-471d-a024-07ce8d383891" (UID: "9322bdc0-598b-471d-a024-07ce8d383891"). InnerVolumeSpecName "kube-api-access-f4cxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:49:34.039465 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.039434    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4cxn\" (UniqueName: \"kubernetes.io/projected/9322bdc0-598b-471d-a024-07ce8d383891-kube-api-access-f4cxn\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:49:34.039465 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.039461    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c212216-7b90-4228-9bf1-cf2718514d4b-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:49:34.039634 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.039476    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dktlv\" (UniqueName: \"kubernetes.io/projected/7c212216-7b90-4228-9bf1-cf2718514d4b-kube-api-access-dktlv\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:49:34.039634 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.039491    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9322bdc0-598b-471d-a024-07ce8d383891-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:49:34.158667 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.158645    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:34.225793 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.225768    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"]
Sep 10 12:49:34.235874 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.235847    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-006b1977-7c8df7c69-lfbp5"]
Sep 10 12:49:34.260422 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.260362    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"]
Sep 10 12:49:34.268563 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:34.268542    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-006b1977-6d79864758-mstqv"]
Sep 10 12:49:36.000739 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:36.000706    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c212216-7b90-4228-9bf1-cf2718514d4b" path="/var/lib/kubelet/pods/7c212216-7b90-4228-9bf1-cf2718514d4b/volumes"
Sep 10 12:49:36.001105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:36.001079    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9322bdc0-598b-471d-a024-07ce8d383891" path="/var/lib/kubelet/pods/9322bdc0-598b-471d-a024-07ce8d383891/volumes"
Sep 10 12:49:43.190488 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:43.190453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:43.333856 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:43.333821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:43.606453 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:43.606406    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:44.036088 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:44.036040    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:44.173696 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:44.173667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:44.282027 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:44.282002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:52.262441 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:52.262352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:52.379730 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:52.379684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:52.533680 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:52.533649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:52.695106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:52.695057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:49:53.431354 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:49:53.431317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:00.310508 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:00.310447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:00.409242 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:00.409212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:01.340295 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:01.340267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:01.841227 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:01.841164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:01.961524 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:01.961489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:03.341098 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:03.341066    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:05.262615 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:05.262557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:05.385408 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:05.385376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:06.717847 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:06.717805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:06.818923 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:06.818890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:07.531418 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.531388    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"]
Sep 10 12:50:07.531855 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.531831    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c212216-7b90-4228-9bf1-cf2718514d4b" containerName="payload-pre-processing"
Sep 10 12:50:07.531855 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.531848    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c212216-7b90-4228-9bf1-cf2718514d4b" containerName="payload-pre-processing"
Sep 10 12:50:07.532099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.531871    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9322bdc0-598b-471d-a024-07ce8d383891" containerName="payload-processing"
Sep 10 12:50:07.532099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.531878    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9322bdc0-598b-471d-a024-07ce8d383891" containerName="payload-processing"
Sep 10 12:50:07.532099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.531946    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9322bdc0-598b-471d-a024-07ce8d383891" containerName="payload-processing"
Sep 10 12:50:07.532099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.531955    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c212216-7b90-4228-9bf1-cf2718514d4b" containerName="payload-pre-processing"
Sep 10 12:50:07.535178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.535159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:07.540103 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.540083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ac8db6\""
Sep 10 12:50:07.567462 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.567440    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"]
Sep 10 12:50:07.580640 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.580620    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:07.616079 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.616054    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"]
Sep 10 12:50:07.619648 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.619632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:07.624949 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:07.624919    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-shared-a-w1-ac8db6-dockercfg-pqjkn\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ac8db6-dockercfg-pqjkn\"" type="*v1.Secret"
Sep 10 12:50:07.627279 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:07.627255    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="acea910d-8e18-4ab3-b607-17130a969fc9" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:07.652048 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.652001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fpp\" (UniqueName: \"kubernetes.io/projected/acea910d-8e18-4ab3-b607-17130a969fc9-kube-api-access-86fpp\") pod \"payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck\" (UID: \"acea910d-8e18-4ab3-b607-17130a969fc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:07.652211 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.652093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mnkg\" (UniqueName: \"kubernetes.io/projected/259e6e80-ae96-48b6-a303-794601fdbdaa-kube-api-access-8mnkg\") pod \"payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r\" (UID: \"259e6e80-ae96-48b6-a303-794601fdbdaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:07.652211 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.652178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acea910d-8e18-4ab3-b607-17130a969fc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck\" (UID: \"acea910d-8e18-4ab3-b607-17130a969fc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:07.652858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.652315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6e80-ae96-48b6-a303-794601fdbdaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r\" (UID: \"259e6e80-ae96-48b6-a303-794601fdbdaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:07.656133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.656107    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"]
Sep 10 12:50:07.700212 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.700189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:07.753044 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.753013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fpp\" (UniqueName: \"kubernetes.io/projected/acea910d-8e18-4ab3-b607-17130a969fc9-kube-api-access-86fpp\") pod \"payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck\" (UID: \"acea910d-8e18-4ab3-b607-17130a969fc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:07.753411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.753079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnkg\" (UniqueName: \"kubernetes.io/projected/259e6e80-ae96-48b6-a303-794601fdbdaa-kube-api-access-8mnkg\") pod \"payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r\" (UID: \"259e6e80-ae96-48b6-a303-794601fdbdaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:07.753411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.753155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acea910d-8e18-4ab3-b607-17130a969fc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck\" (UID: \"acea910d-8e18-4ab3-b607-17130a969fc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:07.753411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.753227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6e80-ae96-48b6-a303-794601fdbdaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r\" (UID: \"259e6e80-ae96-48b6-a303-794601fdbdaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:07.753981 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.753960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acea910d-8e18-4ab3-b607-17130a969fc9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck\" (UID: \"acea910d-8e18-4ab3-b607-17130a969fc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:07.754090 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.754068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6e80-ae96-48b6-a303-794601fdbdaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r\" (UID: \"259e6e80-ae96-48b6-a303-794601fdbdaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:07.766701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.766675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fpp\" (UniqueName: \"kubernetes.io/projected/acea910d-8e18-4ab3-b607-17130a969fc9-kube-api-access-86fpp\") pod \"payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck\" (UID: \"acea910d-8e18-4ab3-b607-17130a969fc9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:07.767815 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.767796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mnkg\" (UniqueName: \"kubernetes.io/projected/259e6e80-ae96-48b6-a303-794601fdbdaa-kube-api-access-8mnkg\") pod \"payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r\" (UID: \"259e6e80-ae96-48b6-a303-794601fdbdaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:07.845720 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:07.845667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:08.133854 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:50:08.133823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259e6e80_ae96_48b6_a303_794601fdbdaa.slice/crio-4b6906753c401225b82e6012324976592ea315cfee40e0f5e2365daedfaf122c WatchSource:0}: Error finding container 4b6906753c401225b82e6012324976592ea315cfee40e0f5e2365daedfaf122c: Status 404 returned error can't find the container with id 4b6906753c401225b82e6012324976592ea315cfee40e0f5e2365daedfaf122c
Sep 10 12:50:08.134425 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:08.134399    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"]
Sep 10 12:50:08.803480 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:08.803450    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ac8db6-dockercfg-pqjkn\""
Sep 10 12:50:08.810550 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:08.810528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:09.100020 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:09.099765    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"]
Sep 10 12:50:09.101419 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:09.101388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r" event={"ID":"259e6e80-ae96-48b6-a303-794601fdbdaa","Type":"ContainerStarted","Data":"0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8"}
Sep 10 12:50:09.101560 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:09.101447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r" event={"ID":"259e6e80-ae96-48b6-a303-794601fdbdaa","Type":"ContainerStarted","Data":"4b6906753c401225b82e6012324976592ea315cfee40e0f5e2365daedfaf122c"}
Sep 10 12:50:09.101671 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:09.101622    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:09.102869 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:50:09.102848    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacea910d_8e18_4ab3_b607_17130a969fc9.slice/crio-2f1d54e9eb1cff198b5a171c8a9ebc41afc03c4938563fcfb6b071811e5ee5ce WatchSource:0}: Error finding container 2f1d54e9eb1cff198b5a171c8a9ebc41afc03c4938563fcfb6b071811e5ee5ce: Status 404 returned error can't find the container with id 2f1d54e9eb1cff198b5a171c8a9ebc41afc03c4938563fcfb6b071811e5ee5ce
Sep 10 12:50:09.143437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:09.143384    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r" podStartSLOduration=2.143364683 podStartE2EDuration="2.143364683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:50:09.14335618 +0000 UTC m=+1183.665525255" watchObservedRunningTime="2026-09-10 12:50:09.143364683 +0000 UTC m=+1183.665533734"
Sep 10 12:50:10.108375 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:10.108337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck" event={"ID":"acea910d-8e18-4ab3-b607-17130a969fc9","Type":"ContainerStarted","Data":"e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223"}
Sep 10 12:50:10.108859 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:10.108395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck" event={"ID":"acea910d-8e18-4ab3-b607-17130a969fc9","Type":"ContainerStarted","Data":"2f1d54e9eb1cff198b5a171c8a9ebc41afc03c4938563fcfb6b071811e5ee5ce"}
Sep 10 12:50:10.108859 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:10.108498    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:10.143825 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:10.143776    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck" podStartSLOduration=3.14376076 podStartE2EDuration="3.14376076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:50:10.1416324 +0000 UTC m=+1184.663801452" watchObservedRunningTime="2026-09-10 12:50:10.14376076 +0000 UTC m=+1184.665929873"
Sep 10 12:50:12.717533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:12.717501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:18.018732 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:18.018694    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"]
Sep 10 12:50:18.028021 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:18.027996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:18.045059 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:18.045036    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-self-signed-certs: secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:18.045153 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:18.045092    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs podName:188a4e7c-bcb0-408e-af38-0f9f5cdbc478 nodeName:}" failed. No retries permitted until 2026-09-10 12:50:18.545078982 +0000 UTC m=+1193.067248012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs") pod "test-model-f85889-kserve-d599c56f-gb9rm" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478") : secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:18.550866 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:18.550830    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-self-signed-certs: secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:18.551032 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:18.550903    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs podName:188a4e7c-bcb0-408e-af38-0f9f5cdbc478 nodeName:}" failed. No retries permitted until 2026-09-10 12:50:19.550887431 +0000 UTC m=+1194.073056461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs") pod "test-model-f85889-kserve-d599c56f-gb9rm" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478") : secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:19.560489 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:19.560456    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-self-signed-certs: secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:19.560976 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:19.560530    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs podName:188a4e7c-bcb0-408e-af38-0f9f5cdbc478 nodeName:}" failed. No retries permitted until 2026-09-10 12:50:21.560512671 +0000 UTC m=+1196.082681700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs") pod "test-model-f85889-kserve-d599c56f-gb9rm" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478") : secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:20.110765 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:20.110727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:50:21.116111 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:21.116077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:50:21.584274 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:21.584240    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-self-signed-certs: secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:21.584434 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:21.584325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs podName:188a4e7c-bcb0-408e-af38-0f9f5cdbc478 nodeName:}" failed. No retries permitted until 2026-09-10 12:50:25.584310497 +0000 UTC m=+1200.106479527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs") pod "test-model-f85889-kserve-d599c56f-gb9rm" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478") : secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:24.744238 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:24.744205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:24.969505 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:24.969471    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:25.080046 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:25.080012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:25.624731 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:25.624699    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-self-signed-certs: secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:25.624904 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:25.624771    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs podName:188a4e7c-bcb0-408e-af38-0f9f5cdbc478 nodeName:}" failed. No retries permitted until 2026-09-10 12:50:33.624753778 +0000 UTC m=+1208.146922818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs") pod "test-model-f85889-kserve-d599c56f-gb9rm" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478") : secret "test-model-f85889-kserve-self-signed-certs" not found
Sep 10 12:50:25.972529 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:25.972445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:50:25.974133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:25.974110    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:50:25.977233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:25.977214    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:50:25.978614 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:25.978576    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:50:26.478768 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:26.478730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:28.535400 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:28.535370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:30.232192 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:30.232162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:30.336457 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:30.336425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:30.622693 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:30.622667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:31.190106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:31.190074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:31.520110 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:31.520078    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:33.169816 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.169764    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm" podUID="188a4e7c-bcb0-408e-af38-0f9f5cdbc478" containerName="main" containerID="cri-o://07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb" gracePeriod=45
Sep 10 12:50:33.423517 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.423464    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:50:33.604734 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.604703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-home\") pod \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") "
Sep 10 12:50:33.604923 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.604763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-dshm\") pod \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") "
Sep 10 12:50:33.604923 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.604806    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvwt2\" (UniqueName: \"kubernetes.io/projected/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-kube-api-access-vvwt2\") pod \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") "
Sep 10 12:50:33.604923 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.604877    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-model-cache\") pod \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") "
Sep 10 12:50:33.605109 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.604955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tmp-dir\") pod \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") "
Sep 10 12:50:33.605109 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.604967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-home" (OuterVolumeSpecName: "home") pod "188a4e7c-bcb0-408e-af38-0f9f5cdbc478" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:33.605109 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.604996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs\") pod \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\" (UID: \"188a4e7c-bcb0-408e-af38-0f9f5cdbc478\") "
Sep 10 12:50:33.605258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.605169    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "188a4e7c-bcb0-408e-af38-0f9f5cdbc478" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:33.605258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.605206    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-model-cache" (OuterVolumeSpecName: "model-cache") pod "188a4e7c-bcb0-408e-af38-0f9f5cdbc478" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:33.605356 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.605338    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:33.605393 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.605362    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:33.605393 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.605375    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:33.607095 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.607071    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-dshm" (OuterVolumeSpecName: "dshm") pod "188a4e7c-bcb0-408e-af38-0f9f5cdbc478" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:33.607214 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.607077    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-kube-api-access-vvwt2" (OuterVolumeSpecName: "kube-api-access-vvwt2") pod "188a4e7c-bcb0-408e-af38-0f9f5cdbc478" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478"). InnerVolumeSpecName "kube-api-access-vvwt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:50:33.607214 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.607098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "188a4e7c-bcb0-408e-af38-0f9f5cdbc478" (UID: "188a4e7c-bcb0-408e-af38-0f9f5cdbc478"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:50:33.706528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.706445    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwt2\" (UniqueName: \"kubernetes.io/projected/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-kube-api-access-vvwt2\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:33.706528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.706477    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:33.706528 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:33.706490    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188a4e7c-bcb0-408e-af38-0f9f5cdbc478-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:34.243836 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.243794    2577 generic.go:356] "Generic (PLEG): container finished" podID="188a4e7c-bcb0-408e-af38-0f9f5cdbc478" containerID="07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb" exitCode=0
Sep 10 12:50:34.243836 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.243835    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"
Sep 10 12:50:34.244300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.243843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm" event={"ID":"188a4e7c-bcb0-408e-af38-0f9f5cdbc478","Type":"ContainerDied","Data":"07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb"}
Sep 10 12:50:34.244300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.243890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm" event={"ID":"188a4e7c-bcb0-408e-af38-0f9f5cdbc478","Type":"ContainerDied","Data":"7457db6555a0b38d14633ea6d4c2eaf17d58743eb3bc326708e4f3dc627bf822"}
Sep 10 12:50:34.244300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.243912    2577 scope.go:117] "RemoveContainer" containerID="07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb"
Sep 10 12:50:34.253105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.253074    2577 scope.go:117] "RemoveContainer" containerID="07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb"
Sep 10 12:50:34.253403 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:34.253382    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb\": container with ID starting with 07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb not found: ID does not exist" containerID="07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb"
Sep 10 12:50:34.253466 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.253412    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb"} err="failed to get container status \"07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb\": rpc error: code = NotFound desc = could not find container \"07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb\": container with ID starting with 07a24a163da394c85fbc313a63d3df4e2c7e8fdc99ad1b91af0df8ef573b51bb not found: ID does not exist"
Sep 10 12:50:34.258509 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.258456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:34.311348 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.311320    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"]
Sep 10 12:50:34.323222 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.323161    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f85889/test-model-f85889-kserve-d599c56f-gb9rm"]
Sep 10 12:50:34.409450 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:34.409414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:35.502039 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:35.502008    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:35.606275 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:35.606249    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:35.755875 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:35.755778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:36.000542 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.000509    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188a4e7c-bcb0-408e-af38-0f9f5cdbc478" path="/var/lib/kubelet/pods/188a4e7c-bcb0-408e-af38-0f9f5cdbc478/volumes"
Sep 10 12:50:36.282493 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.282456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:36.403842 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.403811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:36.510188 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.510149    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"]
Sep 10 12:50:36.510643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.510629    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188a4e7c-bcb0-408e-af38-0f9f5cdbc478" containerName="main"
Sep 10 12:50:36.510643 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.510644    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="188a4e7c-bcb0-408e-af38-0f9f5cdbc478" containerName="main"
Sep 10 12:50:36.510738 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.510706    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="188a4e7c-bcb0-408e-af38-0f9f5cdbc478" containerName="main"
Sep 10 12:50:36.515313 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.515295    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:36.518003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.517985    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-719d8a-dockercfg-pnkdq\""
Sep 10 12:50:36.518094 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.518039    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-719d8a\""
Sep 10 12:50:36.527734 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.527712    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"]
Sep 10 12:50:36.598345 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.598267    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"]
Sep 10 12:50:36.604018 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.603986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:36.614426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.614398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"]
Sep 10 12:50:36.634668 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.634637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs\" (UID: \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:36.634800 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.634708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78sl5\" (UniqueName: \"kubernetes.io/projected/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-kube-api-access-78sl5\") pod \"payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs\" (UID: \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:36.736255 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.736215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs\" (UID: \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:36.736433 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.736291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qdf\" (UniqueName: \"kubernetes.io/projected/cef0f523-e3bf-4101-b1ac-92b1676d00aa-kube-api-access-89qdf\") pod \"payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj\" (UID: \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:36.736433 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.736316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78sl5\" (UniqueName: \"kubernetes.io/projected/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-kube-api-access-78sl5\") pod \"payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs\" (UID: \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:36.736433 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.736347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef0f523-e3bf-4101-b1ac-92b1676d00aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj\" (UID: \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:36.737089 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.737064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs\" (UID: \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:36.747335 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.747310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78sl5\" (UniqueName: \"kubernetes.io/projected/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-kube-api-access-78sl5\") pod \"payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs\" (UID: \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:36.783561 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.783526    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:36.797369 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.797341    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"]
Sep 10 12:50:36.809420 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.809392    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f85889-29817405-zn6np"]
Sep 10 12:50:36.826715 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.826693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:36.837570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.837537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qdf\" (UniqueName: \"kubernetes.io/projected/cef0f523-e3bf-4101-b1ac-92b1676d00aa-kube-api-access-89qdf\") pod \"payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj\" (UID: \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:36.837685 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.837647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef0f523-e3bf-4101-b1ac-92b1676d00aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj\" (UID: \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:36.838562 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.838536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef0f523-e3bf-4101-b1ac-92b1676d00aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj\" (UID: \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:36.851376 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.851312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qdf\" (UniqueName: \"kubernetes.io/projected/cef0f523-e3bf-4101-b1ac-92b1676d00aa-kube-api-access-89qdf\") pod \"payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj\" (UID: \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:36.858169 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.858141    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"]
Sep 10 12:50:36.858668 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.858373    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw" podUID="85729f7d-1f08-4334-982d-aa62ab456d55" containerName="payload-processing" containerID="cri-o://47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911" gracePeriod=30
Sep 10 12:50:36.865486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.865445    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"]
Sep 10 12:50:36.865714 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.865674    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn" podUID="146a34bb-bfc3-4f38-8143-3840fb0983f2" containerName="payload-pre-processing" containerID="cri-o://1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8" gracePeriod=30
Sep 10 12:50:36.913398 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.913323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:36.916083 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:36.915950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:37.011234 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.011150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:37.126624 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.126518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:37.126781 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.126742    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"]
Sep 10 12:50:37.129287 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:50:37.129261    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3644e83e_5927_4c4f_82b8_d87ae6ccc56e.slice/crio-32a5915a67ea0910124da26704b34a2cee4b77bc75b711be58f31eaaa78551a2 WatchSource:0}: Error finding container 32a5915a67ea0910124da26704b34a2cee4b77bc75b711be58f31eaaa78551a2: Status 404 returned error can't find the container with id 32a5915a67ea0910124da26704b34a2cee4b77bc75b711be58f31eaaa78551a2
Sep 10 12:50:37.152063 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.152036    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:50:37.171038 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.171015    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:50:37.208259 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.208233    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"]
Sep 10 12:50:37.210470 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:50:37.210429    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef0f523_e3bf_4101_b1ac_92b1676d00aa.slice/crio-1f4775c40f9060ab63ad6f9a089f968e9a5045dafa45789c28bc3d768d41ce01 WatchSource:0}: Error finding container 1f4775c40f9060ab63ad6f9a089f968e9a5045dafa45789c28bc3d768d41ce01: Status 404 returned error can't find the container with id 1f4775c40f9060ab63ad6f9a089f968e9a5045dafa45789c28bc3d768d41ce01
Sep 10 12:50:37.223132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.223076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:37.267368 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.267342    2577 generic.go:356] "Generic (PLEG): container finished" podID="85729f7d-1f08-4334-982d-aa62ab456d55" containerID="47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911" exitCode=0
Sep 10 12:50:37.267583 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.267380    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"
Sep 10 12:50:37.267583 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.267420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw" event={"ID":"85729f7d-1f08-4334-982d-aa62ab456d55","Type":"ContainerDied","Data":"47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911"}
Sep 10 12:50:37.267583 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.267482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw" event={"ID":"85729f7d-1f08-4334-982d-aa62ab456d55","Type":"ContainerDied","Data":"e946340736a875baf66c259aabdbba7ff0ea2fb574bbba16f3f2b576069f107f"}
Sep 10 12:50:37.267583 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.267517    2577 scope.go:117] "RemoveContainer" containerID="47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911"
Sep 10 12:50:37.268915 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.268893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj" event={"ID":"cef0f523-e3bf-4101-b1ac-92b1676d00aa","Type":"ContainerStarted","Data":"1f4775c40f9060ab63ad6f9a089f968e9a5045dafa45789c28bc3d768d41ce01"}
Sep 10 12:50:37.270460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.270437    2577 generic.go:356] "Generic (PLEG): container finished" podID="146a34bb-bfc3-4f38-8143-3840fb0983f2" containerID="1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8" exitCode=0
Sep 10 12:50:37.270570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.270492    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"
Sep 10 12:50:37.270570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.270526    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn" event={"ID":"146a34bb-bfc3-4f38-8143-3840fb0983f2","Type":"ContainerDied","Data":"1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8"}
Sep 10 12:50:37.270709 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.270580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn" event={"ID":"146a34bb-bfc3-4f38-8143-3840fb0983f2","Type":"ContainerDied","Data":"cb30752048ce3eb669b9aabd515fbe680d135dd8f06cf1b2ad5600fa4ffd5527"}
Sep 10 12:50:37.273323 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.273300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs" event={"ID":"3644e83e-5927-4c4f-82b8-d87ae6ccc56e","Type":"ContainerStarted","Data":"0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d"}
Sep 10 12:50:37.273411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.273349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs" event={"ID":"3644e83e-5927-4c4f-82b8-d87ae6ccc56e","Type":"ContainerStarted","Data":"32a5915a67ea0910124da26704b34a2cee4b77bc75b711be58f31eaaa78551a2"}
Sep 10 12:50:37.273411 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.273382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:37.277876 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.277771    2577 scope.go:117] "RemoveContainer" containerID="47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911"
Sep 10 12:50:37.278115 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:37.278091    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911\": container with ID starting with 47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911 not found: ID does not exist" containerID="47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911"
Sep 10 12:50:37.278199 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.278122    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911"} err="failed to get container status \"47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911\": rpc error: code = NotFound desc = could not find container \"47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911\": container with ID starting with 47d8d7a72bd6c668cd510b7b376e0f6a3842c542bc8eccbc96773d36449aa911 not found: ID does not exist"
Sep 10 12:50:37.278199 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.278144    2577 scope.go:117] "RemoveContainer" containerID="1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8"
Sep 10 12:50:37.286506 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.286491    2577 scope.go:117] "RemoveContainer" containerID="1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8"
Sep 10 12:50:37.286815 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:37.286794    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8\": container with ID starting with 1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8 not found: ID does not exist" containerID="1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8"
Sep 10 12:50:37.286911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.286819    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8"} err="failed to get container status \"1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8\": rpc error: code = NotFound desc = could not find container \"1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8\": container with ID starting with 1da5a7247e92f07ff4af21b994d2005e8255edf56d6ca75c8c0f396bc17fbec8 not found: ID does not exist"
Sep 10 12:50:37.296712 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.296667    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs" podStartSLOduration=1.29665329 podStartE2EDuration="1.29665329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:50:37.295311219 +0000 UTC m=+1211.817480272" watchObservedRunningTime="2026-09-10 12:50:37.29665329 +0000 UTC m=+1211.818822341"
Sep 10 12:50:37.342530 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.342495    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q86vn\" (UniqueName: \"kubernetes.io/projected/146a34bb-bfc3-4f38-8143-3840fb0983f2-kube-api-access-q86vn\") pod \"146a34bb-bfc3-4f38-8143-3840fb0983f2\" (UID: \"146a34bb-bfc3-4f38-8143-3840fb0983f2\") "
Sep 10 12:50:37.342706 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.342620    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146a34bb-bfc3-4f38-8143-3840fb0983f2-plugins-config-volume\") pod \"146a34bb-bfc3-4f38-8143-3840fb0983f2\" (UID: \"146a34bb-bfc3-4f38-8143-3840fb0983f2\") "
Sep 10 12:50:37.342706 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.342673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh5lr\" (UniqueName: \"kubernetes.io/projected/85729f7d-1f08-4334-982d-aa62ab456d55-kube-api-access-dh5lr\") pod \"85729f7d-1f08-4334-982d-aa62ab456d55\" (UID: \"85729f7d-1f08-4334-982d-aa62ab456d55\") "
Sep 10 12:50:37.342833 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.342717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85729f7d-1f08-4334-982d-aa62ab456d55-plugins-config-volume\") pod \"85729f7d-1f08-4334-982d-aa62ab456d55\" (UID: \"85729f7d-1f08-4334-982d-aa62ab456d55\") "
Sep 10 12:50:37.343026 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.342991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146a34bb-bfc3-4f38-8143-3840fb0983f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "146a34bb-bfc3-4f38-8143-3840fb0983f2" (UID: "146a34bb-bfc3-4f38-8143-3840fb0983f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:50:37.343121 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.343105    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146a34bb-bfc3-4f38-8143-3840fb0983f2-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:37.343244 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.343217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85729f7d-1f08-4334-982d-aa62ab456d55-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85729f7d-1f08-4334-982d-aa62ab456d55" (UID: "85729f7d-1f08-4334-982d-aa62ab456d55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:50:37.345094 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.345070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85729f7d-1f08-4334-982d-aa62ab456d55-kube-api-access-dh5lr" (OuterVolumeSpecName: "kube-api-access-dh5lr") pod "85729f7d-1f08-4334-982d-aa62ab456d55" (UID: "85729f7d-1f08-4334-982d-aa62ab456d55"). InnerVolumeSpecName "kube-api-access-dh5lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:50:37.345383 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.345365    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146a34bb-bfc3-4f38-8143-3840fb0983f2-kube-api-access-q86vn" (OuterVolumeSpecName: "kube-api-access-q86vn") pod "146a34bb-bfc3-4f38-8143-3840fb0983f2" (UID: "146a34bb-bfc3-4f38-8143-3840fb0983f2"). InnerVolumeSpecName "kube-api-access-q86vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:50:37.444026 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.443950    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q86vn\" (UniqueName: \"kubernetes.io/projected/146a34bb-bfc3-4f38-8143-3840fb0983f2-kube-api-access-q86vn\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:37.444026 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.443982    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh5lr\" (UniqueName: \"kubernetes.io/projected/85729f7d-1f08-4334-982d-aa62ab456d55-kube-api-access-dh5lr\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:37.444026 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.443996    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85729f7d-1f08-4334-982d-aa62ab456d55-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:37.585279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.585253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:37.602084 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.602055    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"]
Sep 10 12:50:37.621394 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.621366    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f85889-7cb5564b45-s68pw"]
Sep 10 12:50:37.636633 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.636586    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"]
Sep 10 12:50:37.643630 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:37.643582    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f85889-645d65964b-zzwhn"]
Sep 10 12:50:38.001209 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:38.001175    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146a34bb-bfc3-4f38-8143-3840fb0983f2" path="/var/lib/kubelet/pods/146a34bb-bfc3-4f38-8143-3840fb0983f2/volumes"
Sep 10 12:50:38.001478 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:38.001465    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22558955-341f-4c29-968f-7c980369f3a8" path="/var/lib/kubelet/pods/22558955-341f-4c29-968f-7c980369f3a8/volumes"
Sep 10 12:50:38.001771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:38.001758    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85729f7d-1f08-4334-982d-aa62ab456d55" path="/var/lib/kubelet/pods/85729f7d-1f08-4334-982d-aa62ab456d55/volumes"
Sep 10 12:50:38.284141 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:38.284103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj" event={"ID":"cef0f523-e3bf-4101-b1ac-92b1676d00aa","Type":"ContainerStarted","Data":"2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5"}
Sep 10 12:50:38.284444 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:38.284422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:38.311942 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:38.311891    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj" podStartSLOduration=2.311876021 podStartE2EDuration="2.311876021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:50:38.311421097 +0000 UTC m=+1212.833590172" watchObservedRunningTime="2026-09-10 12:50:38.311876021 +0000 UTC m=+1212.834045073"
Sep 10 12:50:40.834033 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:40.834002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:45.003474 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.003440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:45.074091 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.074056    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:45.241645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.237775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:45.332561 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.332529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:45.680888 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.680802    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"]
Sep 10 12:50:45.681282 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.681268    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146a34bb-bfc3-4f38-8143-3840fb0983f2" containerName="payload-pre-processing"
Sep 10 12:50:45.681325 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.681283    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="146a34bb-bfc3-4f38-8143-3840fb0983f2" containerName="payload-pre-processing"
Sep 10 12:50:45.681325 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.681316    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85729f7d-1f08-4334-982d-aa62ab456d55" containerName="payload-processing"
Sep 10 12:50:45.681325 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.681321    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85729f7d-1f08-4334-982d-aa62ab456d55" containerName="payload-processing"
Sep 10 12:50:45.681433 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.681384    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="146a34bb-bfc3-4f38-8143-3840fb0983f2" containerName="payload-pre-processing"
Sep 10 12:50:45.681433 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.681395    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85729f7d-1f08-4334-982d-aa62ab456d55" containerName="payload-processing"
Sep 10 12:50:45.686149 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.686125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"
Sep 10 12:50:45.689122 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.689098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:50:45.700864 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.700837    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"]
Sep 10 12:50:45.722820 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.722781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b66ec9-cd45-4218-a587-159dc0bee109-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b74-j99nm\" (UID: \"76b66ec9-cd45-4218-a587-159dc0bee109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"
Sep 10 12:50:45.824121 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.824087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b66ec9-cd45-4218-a587-159dc0bee109-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b74-j99nm\" (UID: \"76b66ec9-cd45-4218-a587-159dc0bee109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"
Sep 10 12:50:45.824716 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.824695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b66ec9-cd45-4218-a587-159dc0bee109-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b74-j99nm\" (UID: \"76b66ec9-cd45-4218-a587-159dc0bee109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"
Sep 10 12:50:45.829423 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.829400    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:45.996344 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:45.996265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"
Sep 10 12:50:46.272858 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:46.272836    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"]
Sep 10 12:50:46.275563 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:50:46.275535    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b66ec9_cd45_4218_a587_159dc0bee109.slice/crio-92dd5e49a9d7e9f19ac27f566a9ab63fad8e068d659a4a32868915a19427236a WatchSource:0}: Error finding container 92dd5e49a9d7e9f19ac27f566a9ab63fad8e068d659a4a32868915a19427236a: Status 404 returned error can't find the container with id 92dd5e49a9d7e9f19ac27f566a9ab63fad8e068d659a4a32868915a19427236a
Sep 10 12:50:46.327961 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:46.327929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm" event={"ID":"76b66ec9-cd45-4218-a587-159dc0bee109","Type":"ContainerStarted","Data":"92dd5e49a9d7e9f19ac27f566a9ab63fad8e068d659a4a32868915a19427236a"}
Sep 10 12:50:47.334431 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:47.334355    2577 generic.go:356] "Generic (PLEG): container finished" podID="76b66ec9-cd45-4218-a587-159dc0bee109" containerID="12a6926b3c46585ef8f78c7c5370d7d0146835c2916b3277acd724f909f44cb3" exitCode=0
Sep 10 12:50:47.334777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:47.334445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm" event={"ID":"76b66ec9-cd45-4218-a587-159dc0bee109","Type":"ContainerDied","Data":"12a6926b3c46585ef8f78c7c5370d7d0146835c2916b3277acd724f909f44cb3"}
Sep 10 12:50:48.213848 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:48.213802    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"]
Sep 10 12:50:48.223110 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:48.223086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:48.286262 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:48.286235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:50:48.472412 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:48.472355    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"
Sep 10 12:50:48.548656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:48.548629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b66ec9-cd45-4218-a587-159dc0bee109-maas-api-service-ca\") pod \"76b66ec9-cd45-4218-a587-159dc0bee109\" (UID: \"76b66ec9-cd45-4218-a587-159dc0bee109\") "
Sep 10 12:50:48.548998 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:48.548974    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b66ec9-cd45-4218-a587-159dc0bee109-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76b66ec9-cd45-4218-a587-159dc0bee109" (UID: "76b66ec9-cd45-4218-a587-159dc0bee109"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:50:48.650239 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:48.650203    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b66ec9-cd45-4218-a587-159dc0bee109-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:49.291801 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:49.291759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:50:49.348398 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:49.348374    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"
Sep 10 12:50:49.348557 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:49.348421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm" event={"ID":"76b66ec9-cd45-4218-a587-159dc0bee109","Type":"ContainerDied","Data":"92dd5e49a9d7e9f19ac27f566a9ab63fad8e068d659a4a32868915a19427236a"}
Sep 10 12:50:49.348557 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:49.348481    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92dd5e49a9d7e9f19ac27f566a9ab63fad8e068d659a4a32868915a19427236a"
Sep 10 12:50:52.028706 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:52.028675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:55.982588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:55.982547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:57.321281 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.321250    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:57.877189 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.877150    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"]
Sep 10 12:50:57.880191 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.880166    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-714b74-29817405-mnvdp"]
Sep 10 12:50:57.882896 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.882873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:57.916000 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.915971    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"]
Sep 10 12:50:57.916490 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.916460    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x" podUID="67869779-0a1b-455e-bea2-d96fec6c268e" containerName="payload-processing" containerID="cri-o://9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60" gracePeriod=30
Sep 10 12:50:57.920131 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.920053    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"]
Sep 10 12:50:57.920306 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.920281    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n" podUID="6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" containerName="payload-pre-processing" containerID="cri-o://b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f" gracePeriod=30
Sep 10 12:50:57.966003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:57.965972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:58.001331 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.001307    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f24409d2-4fa9-4caf-bca3-710bb9539119" path="/var/lib/kubelet/pods/f24409d2-4fa9-4caf-bca3-710bb9539119/volumes"
Sep 10 12:50:58.036667 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.036642    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:58.137758 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.137695    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:58.140352 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.140326    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"]
Sep 10 12:50:58.148490 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.148460    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b74-j99nm"]
Sep 10 12:50:58.199937 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.199915    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:50:58.203084 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.203066    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:50:58.213145 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.213125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:58.235943 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.235918    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrqzm\" (UniqueName: \"kubernetes.io/projected/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-kube-api-access-vrqzm\") pod \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\" (UID: \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\") "
Sep 10 12:50:58.236041 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.235952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk94p\" (UniqueName: \"kubernetes.io/projected/67869779-0a1b-455e-bea2-d96fec6c268e-kube-api-access-vk94p\") pod \"67869779-0a1b-455e-bea2-d96fec6c268e\" (UID: \"67869779-0a1b-455e-bea2-d96fec6c268e\") "
Sep 10 12:50:58.236041 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.236008    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67869779-0a1b-455e-bea2-d96fec6c268e-plugins-config-volume\") pod \"67869779-0a1b-455e-bea2-d96fec6c268e\" (UID: \"67869779-0a1b-455e-bea2-d96fec6c268e\") "
Sep 10 12:50:58.236219 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.236050    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-plugins-config-volume\") pod \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\" (UID: \"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1\") "
Sep 10 12:50:58.236431 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.236402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67869779-0a1b-455e-bea2-d96fec6c268e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67869779-0a1b-455e-bea2-d96fec6c268e" (UID: "67869779-0a1b-455e-bea2-d96fec6c268e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:50:58.236508 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.236486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" (UID: "6f7d1cf6-d18f-40ad-a6a3-1188c3350da1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:50:58.238188 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.238164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-kube-api-access-vrqzm" (OuterVolumeSpecName: "kube-api-access-vrqzm") pod "6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" (UID: "6f7d1cf6-d18f-40ad-a6a3-1188c3350da1"). InnerVolumeSpecName "kube-api-access-vrqzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:50:58.238288 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.238194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67869779-0a1b-455e-bea2-d96fec6c268e-kube-api-access-vk94p" (OuterVolumeSpecName: "kube-api-access-vk94p") pod "67869779-0a1b-455e-bea2-d96fec6c268e" (UID: "67869779-0a1b-455e-bea2-d96fec6c268e"). InnerVolumeSpecName "kube-api-access-vk94p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:50:58.337799 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.337761    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrqzm\" (UniqueName: \"kubernetes.io/projected/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-kube-api-access-vrqzm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:58.337799 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.337792    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk94p\" (UniqueName: \"kubernetes.io/projected/67869779-0a1b-455e-bea2-d96fec6c268e-kube-api-access-vk94p\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:58.337799 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.337806    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67869779-0a1b-455e-bea2-d96fec6c268e-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:58.338214 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.337818    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:50:58.401533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.401451    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" containerID="b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f" exitCode=0
Sep 10 12:50:58.401533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.401506    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"
Sep 10 12:50:58.401775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.401570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n" event={"ID":"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1","Type":"ContainerDied","Data":"b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f"}
Sep 10 12:50:58.401775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.401644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n" event={"ID":"6f7d1cf6-d18f-40ad-a6a3-1188c3350da1","Type":"ContainerDied","Data":"e041b6f62055996bd1b5cac7a6f212902446a1a7e8944cea62834bc024c5536b"}
Sep 10 12:50:58.401775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.401671    2577 scope.go:117] "RemoveContainer" containerID="b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f"
Sep 10 12:50:58.403009 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.402986    2577 generic.go:356] "Generic (PLEG): container finished" podID="67869779-0a1b-455e-bea2-d96fec6c268e" containerID="9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60" exitCode=0
Sep 10 12:50:58.403124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.403019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x" event={"ID":"67869779-0a1b-455e-bea2-d96fec6c268e","Type":"ContainerDied","Data":"9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60"}
Sep 10 12:50:58.403124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.403075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x" event={"ID":"67869779-0a1b-455e-bea2-d96fec6c268e","Type":"ContainerDied","Data":"58ab1e86cf9c1471ecc297270267cbd7dadecef68fee5a1a8487a4fcd81eacb5"}
Sep 10 12:50:58.403124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.403036    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"
Sep 10 12:50:58.411430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.411415    2577 scope.go:117] "RemoveContainer" containerID="b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f"
Sep 10 12:50:58.411729 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:58.411712    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f\": container with ID starting with b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f not found: ID does not exist" containerID="b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f"
Sep 10 12:50:58.411800 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.411736    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f"} err="failed to get container status \"b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f\": rpc error: code = NotFound desc = could not find container \"b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f\": container with ID starting with b2de0b0066c7e715129672261a6ca36e70aede2be69a1b136b49e780ebbbc26f not found: ID does not exist"
Sep 10 12:50:58.411800 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.411751    2577 scope.go:117] "RemoveContainer" containerID="9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60"
Sep 10 12:50:58.419693 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.419675    2577 scope.go:117] "RemoveContainer" containerID="9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60"
Sep 10 12:50:58.419962 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:50:58.419942    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60\": container with ID starting with 9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60 not found: ID does not exist" containerID="9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60"
Sep 10 12:50:58.420032 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.419972    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60"} err="failed to get container status \"9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60\": rpc error: code = NotFound desc = could not find container \"9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60\": container with ID starting with 9efa978a23a8532ddb31d12928fb0532c9f76990f34045df068a37dad4948f60 not found: ID does not exist"
Sep 10 12:50:58.428158 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.428138    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"]
Sep 10 12:50:58.431304 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.431282    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-714b74-74dc7b979c-wf86x"]
Sep 10 12:50:58.444636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.444613    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"]
Sep 10 12:50:58.450139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.450119    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-714b74-869588ff74-dwc6n"]
Sep 10 12:50:58.496853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:58.496821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:50:59.187466 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:50:59.187436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:00.000717 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:00.000682    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67869779-0a1b-455e-bea2-d96fec6c268e" path="/var/lib/kubelet/pods/67869779-0a1b-455e-bea2-d96fec6c268e/volumes"
Sep 10 12:51:00.001080 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:00.001068    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" path="/var/lib/kubelet/pods/6f7d1cf6-d18f-40ad-a6a3-1188c3350da1/volumes"
Sep 10 12:51:00.001356 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:00.001343    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b66ec9-cd45-4218-a587-159dc0bee109" path="/var/lib/kubelet/pods/76b66ec9-cd45-4218-a587-159dc0bee109/volumes"
Sep 10 12:51:03.233500 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.233459    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4" podUID="90b1403c-dede-4b7d-aca2-8d617d82d610" containerName="main" containerID="cri-o://bc205e6e96383af0c830e13b5768423c67726978753cc3ae9160d0af09044bcf" gracePeriod=45
Sep 10 12:51:03.258182 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258157    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-e2b4c0"]
Sep 10 12:51:03.258659 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258642    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" containerName="payload-pre-processing"
Sep 10 12:51:03.258747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258660    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" containerName="payload-pre-processing"
Sep 10 12:51:03.258747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258683    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b66ec9-cd45-4218-a587-159dc0bee109" containerName="revoke-keys"
Sep 10 12:51:03.258747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258692    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b66ec9-cd45-4218-a587-159dc0bee109" containerName="revoke-keys"
Sep 10 12:51:03.258747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258717    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67869779-0a1b-455e-bea2-d96fec6c268e" containerName="payload-processing"
Sep 10 12:51:03.258747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258726    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67869779-0a1b-455e-bea2-d96fec6c268e" containerName="payload-processing"
Sep 10 12:51:03.258997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258836    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7d1cf6-d18f-40ad-a6a3-1188c3350da1" containerName="payload-pre-processing"
Sep 10 12:51:03.258997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258850    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="67869779-0a1b-455e-bea2-d96fec6c268e" containerName="payload-processing"
Sep 10 12:51:03.258997 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.258860    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b66ec9-cd45-4218-a587-159dc0bee109" containerName="revoke-keys"
Sep 10 12:51:03.261922 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.261905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-e2b4c0"
Sep 10 12:51:03.264486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.264467    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-w7tsg\""
Sep 10 12:51:03.266961 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.266940    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-e2b4c0"]
Sep 10 12:51:03.382479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.382449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbljd\" (UniqueName: \"kubernetes.io/projected/bd896b46-0b66-4fe8-89cd-9ebde5677400-kube-api-access-sbljd\") pod \"test-curl-7701-e2b4c0\" (UID: \"bd896b46-0b66-4fe8-89cd-9ebde5677400\") " pod="opendatahub/test-curl-7701-e2b4c0"
Sep 10 12:51:03.435271 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.435241    2577 generic.go:356] "Generic (PLEG): container finished" podID="90b1403c-dede-4b7d-aca2-8d617d82d610" containerID="bc205e6e96383af0c830e13b5768423c67726978753cc3ae9160d0af09044bcf" exitCode=0
Sep 10 12:51:03.435412 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.435312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4" event={"ID":"90b1403c-dede-4b7d-aca2-8d617d82d610","Type":"ContainerDied","Data":"bc205e6e96383af0c830e13b5768423c67726978753cc3ae9160d0af09044bcf"}
Sep 10 12:51:03.480102 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.480079    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:51:03.483464 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.483442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbljd\" (UniqueName: \"kubernetes.io/projected/bd896b46-0b66-4fe8-89cd-9ebde5677400-kube-api-access-sbljd\") pod \"test-curl-7701-e2b4c0\" (UID: \"bd896b46-0b66-4fe8-89cd-9ebde5677400\") " pod="opendatahub/test-curl-7701-e2b4c0"
Sep 10 12:51:03.491221 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.491174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbljd\" (UniqueName: \"kubernetes.io/projected/bd896b46-0b66-4fe8-89cd-9ebde5677400-kube-api-access-sbljd\") pod \"test-curl-7701-e2b4c0\" (UID: \"bd896b46-0b66-4fe8-89cd-9ebde5677400\") " pod="opendatahub/test-curl-7701-e2b4c0"
Sep 10 12:51:03.528539 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.528514    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:03.584719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.584698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-home\") pod \"90b1403c-dede-4b7d-aca2-8d617d82d610\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") "
Sep 10 12:51:03.584816 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.584795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-model-cache\") pod \"90b1403c-dede-4b7d-aca2-8d617d82d610\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") "
Sep 10 12:51:03.584816 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.584813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-dshm\") pod \"90b1403c-dede-4b7d-aca2-8d617d82d610\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") "
Sep 10 12:51:03.584903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.584835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b1403c-dede-4b7d-aca2-8d617d82d610-tls-certs\") pod \"90b1403c-dede-4b7d-aca2-8d617d82d610\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") "
Sep 10 12:51:03.584903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.584861    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-tmp-dir\") pod \"90b1403c-dede-4b7d-aca2-8d617d82d610\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") "
Sep 10 12:51:03.584992 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.584918    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kkh4\" (UniqueName: \"kubernetes.io/projected/90b1403c-dede-4b7d-aca2-8d617d82d610-kube-api-access-9kkh4\") pod \"90b1403c-dede-4b7d-aca2-8d617d82d610\" (UID: \"90b1403c-dede-4b7d-aca2-8d617d82d610\") "
Sep 10 12:51:03.584992 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.584960    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-home" (OuterVolumeSpecName: "home") pod "90b1403c-dede-4b7d-aca2-8d617d82d610" (UID: "90b1403c-dede-4b7d-aca2-8d617d82d610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:03.585147 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.585126    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-model-cache" (OuterVolumeSpecName: "model-cache") pod "90b1403c-dede-4b7d-aca2-8d617d82d610" (UID: "90b1403c-dede-4b7d-aca2-8d617d82d610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:03.585234 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.585193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90b1403c-dede-4b7d-aca2-8d617d82d610" (UID: "90b1403c-dede-4b7d-aca2-8d617d82d610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:03.585311 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.585230    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:03.585311 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.585245    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:03.586881 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.586858    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b1403c-dede-4b7d-aca2-8d617d82d610-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90b1403c-dede-4b7d-aca2-8d617d82d610" (UID: "90b1403c-dede-4b7d-aca2-8d617d82d610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:51:03.586986 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.586909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-dshm" (OuterVolumeSpecName: "dshm") pod "90b1403c-dede-4b7d-aca2-8d617d82d610" (UID: "90b1403c-dede-4b7d-aca2-8d617d82d610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:03.586986 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.586971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b1403c-dede-4b7d-aca2-8d617d82d610-kube-api-access-9kkh4" (OuterVolumeSpecName: "kube-api-access-9kkh4") pod "90b1403c-dede-4b7d-aca2-8d617d82d610" (UID: "90b1403c-dede-4b7d-aca2-8d617d82d610"). InnerVolumeSpecName "kube-api-access-9kkh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:51:03.608124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.608090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-e2b4c0"
Sep 10 12:51:03.686407 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.686375    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:03.686407 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.686405    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b1403c-dede-4b7d-aca2-8d617d82d610-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:03.686570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.686415    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b1403c-dede-4b7d-aca2-8d617d82d610-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:03.686570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.686424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kkh4\" (UniqueName: \"kubernetes.io/projected/90b1403c-dede-4b7d-aca2-8d617d82d610-kube-api-access-9kkh4\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:03.877285 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:03.877262    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-e2b4c0"]
Sep 10 12:51:03.879366 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:03.879332    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd896b46_0b66_4fe8_89cd_9ebde5677400.slice/crio-9a945c9f24a1baa560fcec399b1e59e023f64368f5ccac78b01d341b14cbbb4a WatchSource:0}: Error finding container 9a945c9f24a1baa560fcec399b1e59e023f64368f5ccac78b01d341b14cbbb4a: Status 404 returned error can't find the container with id 9a945c9f24a1baa560fcec399b1e59e023f64368f5ccac78b01d341b14cbbb4a
Sep 10 12:51:04.442387 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:04.442357    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"
Sep 10 12:51:04.442854 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:04.442360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4" event={"ID":"90b1403c-dede-4b7d-aca2-8d617d82d610","Type":"ContainerDied","Data":"e48f15636a7603a34a6fd586868fc683572ca57eb4fa3e59be14b9b2c3122dcd"}
Sep 10 12:51:04.442854 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:04.442494    2577 scope.go:117] "RemoveContainer" containerID="bc205e6e96383af0c830e13b5768423c67726978753cc3ae9160d0af09044bcf"
Sep 10 12:51:04.444163 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:04.444138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-e2b4c0" event={"ID":"bd896b46-0b66-4fe8-89cd-9ebde5677400","Type":"ContainerStarted","Data":"ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1"}
Sep 10 12:51:04.444270 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:04.444173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-e2b4c0" event={"ID":"bd896b46-0b66-4fe8-89cd-9ebde5677400","Type":"ContainerStarted","Data":"9a945c9f24a1baa560fcec399b1e59e023f64368f5ccac78b01d341b14cbbb4a"}
Sep 10 12:51:04.464172 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:04.464134    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7701-e2b4c0" podStartSLOduration=1.464122095 podStartE2EDuration="1.464122095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:04.461572812 +0000 UTC m=+1238.983741864" watchObservedRunningTime="2026-09-10 12:51:04.464122095 +0000 UTC m=+1238.986291146"
Sep 10 12:51:04.478346 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:04.478325    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"]
Sep 10 12:51:04.483690 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:04.483670    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-714b74/test-model-714b74-kserve-57f898f5ff-tgqs4"]
Sep 10 12:51:05.108745 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.108709    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-e2b4c0"]
Sep 10 12:51:05.113943 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.113874    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-e2b4c0"]
Sep 10 12:51:05.226249 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.226220    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-0dbef9"]
Sep 10 12:51:05.226680 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.226666    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd896b46-0b66-4fe8-89cd-9ebde5677400" containerName="test-curl-7701-e2b4c0"
Sep 10 12:51:05.226746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.226682    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd896b46-0b66-4fe8-89cd-9ebde5677400" containerName="test-curl-7701-e2b4c0"
Sep 10 12:51:05.226746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.226703    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b1403c-dede-4b7d-aca2-8d617d82d610" containerName="main"
Sep 10 12:51:05.226746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.226708    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b1403c-dede-4b7d-aca2-8d617d82d610" containerName="main"
Sep 10 12:51:05.226853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.226774    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b1403c-dede-4b7d-aca2-8d617d82d610" containerName="main"
Sep 10 12:51:05.226853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.226787    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd896b46-0b66-4fe8-89cd-9ebde5677400" containerName="test-curl-7701-e2b4c0"
Sep 10 12:51:05.231129 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.231113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-0dbef9"
Sep 10 12:51:05.234430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.234405    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-0dbef9"]
Sep 10 12:51:05.301021 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.300989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbw9v\" (UniqueName: \"kubernetes.io/projected/67d5cb5b-7640-4da5-a86b-a8cfba0be954-kube-api-access-mbw9v\") pod \"test-curl-7701-0dbef9\" (UID: \"67d5cb5b-7640-4da5-a86b-a8cfba0be954\") " pod="opendatahub/test-curl-7701-0dbef9"
Sep 10 12:51:05.402037 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.401952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbw9v\" (UniqueName: \"kubernetes.io/projected/67d5cb5b-7640-4da5-a86b-a8cfba0be954-kube-api-access-mbw9v\") pod \"test-curl-7701-0dbef9\" (UID: \"67d5cb5b-7640-4da5-a86b-a8cfba0be954\") " pod="opendatahub/test-curl-7701-0dbef9"
Sep 10 12:51:05.410254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.410230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbw9v\" (UniqueName: \"kubernetes.io/projected/67d5cb5b-7640-4da5-a86b-a8cfba0be954-kube-api-access-mbw9v\") pod \"test-curl-7701-0dbef9\" (UID: \"67d5cb5b-7640-4da5-a86b-a8cfba0be954\") " pod="opendatahub/test-curl-7701-0dbef9"
Sep 10 12:51:05.451286 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:05.451264    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7701-e2b4c0" container="test-curl-7701-e2b4c0"
Sep 10 12:51:05.542506 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.542481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-0dbef9"
Sep 10 12:51:05.812328 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:05.812305    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-0dbef9"]
Sep 10 12:51:05.813936 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:05.813909    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67d5cb5b_7640_4da5_a86b_a8cfba0be954.slice/crio-cdc1e983f82cd7c45ec0c2276e6260235a7296ff447edd0666b95da68c9608ca WatchSource:0}: Error finding container cdc1e983f82cd7c45ec0c2276e6260235a7296ff447edd0666b95da68c9608ca: Status 404 returned error can't find the container with id cdc1e983f82cd7c45ec0c2276e6260235a7296ff447edd0666b95da68c9608ca
Sep 10 12:51:06.000054 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:06.000025    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b1403c-dede-4b7d-aca2-8d617d82d610" path="/var/lib/kubelet/pods/90b1403c-dede-4b7d-aca2-8d617d82d610/volumes"
Sep 10 12:51:06.458206 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:06.458162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-0dbef9" event={"ID":"67d5cb5b-7640-4da5-a86b-a8cfba0be954","Type":"ContainerStarted","Data":"788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de"}
Sep 10 12:51:06.458206 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:06.458210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-0dbef9" event={"ID":"67d5cb5b-7640-4da5-a86b-a8cfba0be954","Type":"ContainerStarted","Data":"cdc1e983f82cd7c45ec0c2276e6260235a7296ff447edd0666b95da68c9608ca"}
Sep 10 12:51:06.458727 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:06.458288    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7701-e2b4c0" podUID="bd896b46-0b66-4fe8-89cd-9ebde5677400" containerName="test-curl-7701-e2b4c0" containerID="cri-o://ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1" gracePeriod=2
Sep 10 12:51:06.477589 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:06.477072    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7701-0dbef9" podStartSLOduration=1.477052888 podStartE2EDuration="1.477052888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:06.47419326 +0000 UTC m=+1240.996362317" watchObservedRunningTime="2026-09-10 12:51:06.477052888 +0000 UTC m=+1240.999221943"
Sep 10 12:51:07.077407 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.077368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-0dbef9"]
Sep 10 12:51:07.080281 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.080254    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-0dbef9"]
Sep 10 12:51:07.198064 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.197989    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-009325"]
Sep 10 12:51:07.198417 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.198404    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d5cb5b-7640-4da5-a86b-a8cfba0be954" containerName="test-curl-7701-0dbef9"
Sep 10 12:51:07.198460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.198417    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d5cb5b-7640-4da5-a86b-a8cfba0be954" containerName="test-curl-7701-0dbef9"
Sep 10 12:51:07.198495 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.198477    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d5cb5b-7640-4da5-a86b-a8cfba0be954" containerName="test-curl-7701-0dbef9"
Sep 10 12:51:07.201786 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.201771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-009325"
Sep 10 12:51:07.209703 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.209678    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-009325"]
Sep 10 12:51:07.319823 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.319786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/353f4715-d8b9-41ad-8316-e99b366c4a0f-kube-api-access-wf55w\") pod \"test-curl-7701-009325\" (UID: \"353f4715-d8b9-41ad-8316-e99b366c4a0f\") " pod="opendatahub/test-curl-7701-009325"
Sep 10 12:51:07.421103 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.421071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/353f4715-d8b9-41ad-8316-e99b366c4a0f-kube-api-access-wf55w\") pod \"test-curl-7701-009325\" (UID: \"353f4715-d8b9-41ad-8316-e99b366c4a0f\") " pod="opendatahub/test-curl-7701-009325"
Sep 10 12:51:07.429926 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.429897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/353f4715-d8b9-41ad-8316-e99b366c4a0f-kube-api-access-wf55w\") pod \"test-curl-7701-009325\" (UID: \"353f4715-d8b9-41ad-8316-e99b366c4a0f\") " pod="opendatahub/test-curl-7701-009325"
Sep 10 12:51:07.465085 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:07.465028    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7701-0dbef9" container="test-curl-7701-0dbef9"
Sep 10 12:51:07.511679 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.511658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-009325"
Sep 10 12:51:07.787104 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:07.787074    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-009325"]
Sep 10 12:51:07.790489 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:07.790463    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod353f4715_d8b9_41ad_8316_e99b366c4a0f.slice/crio-cec298a6020d5f7115520f47d13b84f49524416d1fc9a7573712f041c9022a7f WatchSource:0}: Error finding container cec298a6020d5f7115520f47d13b84f49524416d1fc9a7573712f041c9022a7f: Status 404 returned error can't find the container with id cec298a6020d5f7115520f47d13b84f49524416d1fc9a7573712f041c9022a7f
Sep 10 12:51:08.252191 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.252102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:08.472364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.472319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-009325" event={"ID":"353f4715-d8b9-41ad-8316-e99b366c4a0f","Type":"ContainerStarted","Data":"660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f"}
Sep 10 12:51:08.472735 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.472382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-009325" event={"ID":"353f4715-d8b9-41ad-8316-e99b366c4a0f","Type":"ContainerStarted","Data":"cec298a6020d5f7115520f47d13b84f49524416d1fc9a7573712f041c9022a7f"}
Sep 10 12:51:08.472735 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.472480    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7701-0dbef9" podUID="67d5cb5b-7640-4da5-a86b-a8cfba0be954" containerName="test-curl-7701-0dbef9" containerID="cri-o://788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de" gracePeriod=2
Sep 10 12:51:08.487438 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.487293    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7701-009325" podStartSLOduration=1.487275812 podStartE2EDuration="1.487275812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:08.486810908 +0000 UTC m=+1243.008979959" watchObservedRunningTime="2026-09-10 12:51:08.487275812 +0000 UTC m=+1243.009444865"
Sep 10 12:51:08.647851 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.647828    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-e2b4c0"
Sep 10 12:51:08.734887 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.734860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbljd\" (UniqueName: \"kubernetes.io/projected/bd896b46-0b66-4fe8-89cd-9ebde5677400-kube-api-access-sbljd\") pod \"bd896b46-0b66-4fe8-89cd-9ebde5677400\" (UID: \"bd896b46-0b66-4fe8-89cd-9ebde5677400\") "
Sep 10 12:51:08.736810 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.736785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd896b46-0b66-4fe8-89cd-9ebde5677400-kube-api-access-sbljd" (OuterVolumeSpecName: "kube-api-access-sbljd") pod "bd896b46-0b66-4fe8-89cd-9ebde5677400" (UID: "bd896b46-0b66-4fe8-89cd-9ebde5677400"). InnerVolumeSpecName "kube-api-access-sbljd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:51:08.836234 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:08.836205    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbljd\" (UniqueName: \"kubernetes.io/projected/bd896b46-0b66-4fe8-89cd-9ebde5677400-kube-api-access-sbljd\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:09.091127 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.091022    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-009325"]
Sep 10 12:51:09.096861 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.096832    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-009325"]
Sep 10 12:51:09.480061 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.479978    2577 generic.go:356] "Generic (PLEG): container finished" podID="bd896b46-0b66-4fe8-89cd-9ebde5677400" containerID="ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1" exitCode=137
Sep 10 12:51:09.480061 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.480037    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-e2b4c0"
Sep 10 12:51:09.480522 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.480042    2577 scope.go:117] "RemoveContainer" containerID="ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1"
Sep 10 12:51:09.480522 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:09.480427    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7701-009325" container="test-curl-7701-009325"
Sep 10 12:51:09.489137 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.489121    2577 scope.go:117] "RemoveContainer" containerID="ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1"
Sep 10 12:51:09.489413 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:09.489395    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1\": container with ID starting with ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1 not found: ID does not exist" containerID="ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1"
Sep 10 12:51:09.489471 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.489422    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1"} err="failed to get container status \"ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1\": rpc error: code = NotFound desc = could not find container \"ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1\": container with ID starting with ea50c44139692258e381191fc8fcbf01d9a4941a3ce4e44e490add6357af84d1 not found: ID does not exist"
Sep 10 12:51:09.490222 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:09.490199    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-e2b4c0\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="bd896b46-0b66-4fe8-89cd-9ebde5677400" pod="opendatahub/test-curl-7701-e2b4c0"
Sep 10 12:51:09.686017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.685987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:09.794210 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:09.794181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:10.001995 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.001965    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd896b46-0b66-4fe8-89cd-9ebde5677400" path="/var/lib/kubelet/pods/bd896b46-0b66-4fe8-89cd-9ebde5677400/volumes"
Sep 10 12:51:10.050960 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.050891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:10.233342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.233319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:10.487131 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.487095    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7701-009325" podUID="353f4715-d8b9-41ad-8316-e99b366c4a0f" containerName="test-curl-7701-009325" containerID="cri-o://660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f" gracePeriod=2
Sep 10 12:51:10.522917 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.522887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:10.594114 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.594087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:10.617493 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.617470    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-0dbef9"
Sep 10 12:51:10.754016 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.753859    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbw9v\" (UniqueName: \"kubernetes.io/projected/67d5cb5b-7640-4da5-a86b-a8cfba0be954-kube-api-access-mbw9v\") pod \"67d5cb5b-7640-4da5-a86b-a8cfba0be954\" (UID: \"67d5cb5b-7640-4da5-a86b-a8cfba0be954\") "
Sep 10 12:51:10.756420 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.756394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d5cb5b-7640-4da5-a86b-a8cfba0be954-kube-api-access-mbw9v" (OuterVolumeSpecName: "kube-api-access-mbw9v") pod "67d5cb5b-7640-4da5-a86b-a8cfba0be954" (UID: "67d5cb5b-7640-4da5-a86b-a8cfba0be954"). InnerVolumeSpecName "kube-api-access-mbw9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:51:10.855218 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.855189    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbw9v\" (UniqueName: \"kubernetes.io/projected/67d5cb5b-7640-4da5-a86b-a8cfba0be954-kube-api-access-mbw9v\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:10.871934 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:10.871903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:11.065985 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:11.065946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:11.494315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:11.494243    2577 generic.go:356] "Generic (PLEG): container finished" podID="67d5cb5b-7640-4da5-a86b-a8cfba0be954" containerID="788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de" exitCode=137
Sep 10 12:51:11.494315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:11.494288    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-0dbef9"
Sep 10 12:51:11.494315 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:11.494308    2577 scope.go:117] "RemoveContainer" containerID="788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de"
Sep 10 12:51:11.504701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:11.504589    2577 scope.go:117] "RemoveContainer" containerID="788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de"
Sep 10 12:51:11.505090 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:11.505062    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de\": container with ID starting with 788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de not found: ID does not exist" containerID="788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de"
Sep 10 12:51:11.505167 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:11.505100    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-0dbef9\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="67d5cb5b-7640-4da5-a86b-a8cfba0be954" pod="opendatahub/test-curl-7701-0dbef9"
Sep 10 12:51:11.505167 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:11.505100    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de"} err="failed to get container status \"788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de\": rpc error: code = NotFound desc = could not find container \"788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de\": container with ID starting with 788f59fc36e13444976b45dbb00d5e4ac1111c203c248fe8fcf77192608566de not found: ID does not exist"
Sep 10 12:51:12.000410 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:12.000382    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d5cb5b-7640-4da5-a86b-a8cfba0be954" path="/var/lib/kubelet/pods/67d5cb5b-7640-4da5-a86b-a8cfba0be954/volumes"
Sep 10 12:51:12.663924 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:12.663898    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-009325"
Sep 10 12:51:12.773132 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:12.773108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/353f4715-d8b9-41ad-8316-e99b366c4a0f-kube-api-access-wf55w\") pod \"353f4715-d8b9-41ad-8316-e99b366c4a0f\" (UID: \"353f4715-d8b9-41ad-8316-e99b366c4a0f\") "
Sep 10 12:51:12.775091 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:12.775057    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353f4715-d8b9-41ad-8316-e99b366c4a0f-kube-api-access-wf55w" (OuterVolumeSpecName: "kube-api-access-wf55w") pod "353f4715-d8b9-41ad-8316-e99b366c4a0f" (UID: "353f4715-d8b9-41ad-8316-e99b366c4a0f"). InnerVolumeSpecName "kube-api-access-wf55w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:51:12.874051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:12.874020    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf55w\" (UniqueName: \"kubernetes.io/projected/353f4715-d8b9-41ad-8316-e99b366c4a0f-kube-api-access-wf55w\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:51:13.509542 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:13.509508    2577 generic.go:356] "Generic (PLEG): container finished" podID="353f4715-d8b9-41ad-8316-e99b366c4a0f" containerID="660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f" exitCode=137
Sep 10 12:51:13.509542 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:13.509544    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-009325"
Sep 10 12:51:13.509813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:13.509613    2577 scope.go:117] "RemoveContainer" containerID="660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f"
Sep 10 12:51:13.520320 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:13.520165    2577 scope.go:117] "RemoveContainer" containerID="660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f"
Sep 10 12:51:13.520541 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:13.520521    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f\": container with ID starting with 660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f not found: ID does not exist" containerID="660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f"
Sep 10 12:51:13.520620 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:13.520548    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f"} err="failed to get container status \"660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f\": rpc error: code = NotFound desc = could not find container \"660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f\": container with ID starting with 660fb2b19d7d974dff6a550d57712ae93558b21ef30c2ec728570f9029b2661f not found: ID does not exist"
Sep 10 12:51:13.521701 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:51:13.521657    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-009325\" is forbidden: User \"system:node:ip-10-0-140-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-169.ec2.internal' and this object" podUID="353f4715-d8b9-41ad-8316-e99b366c4a0f" pod="opendatahub/test-curl-7701-009325"
Sep 10 12:51:13.556703 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:13.556678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:13.683685 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:13.683657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:13.792308 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:13.792282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:14.000440 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.000406    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="353f4715-d8b9-41ad-8316-e99b366c4a0f" path="/var/lib/kubelet/pods/353f4715-d8b9-41ad-8316-e99b366c4a0f/volumes"
Sep 10 12:51:14.505054 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.505017    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"]
Sep 10 12:51:14.505453 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.505432    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353f4715-d8b9-41ad-8316-e99b366c4a0f" containerName="test-curl-7701-009325"
Sep 10 12:51:14.505453 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.505446    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="353f4715-d8b9-41ad-8316-e99b366c4a0f" containerName="test-curl-7701-009325"
Sep 10 12:51:14.505636 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.505517    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="353f4715-d8b9-41ad-8316-e99b366c4a0f" containerName="test-curl-7701-009325"
Sep 10 12:51:14.510334 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.510259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.515200 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.515112    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"default-dockercfg-8xhp5\""
Sep 10 12:51:14.515717 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.515455    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"distinct-2-w0-30919a-kserve-self-signed-certs\""
Sep 10 12:51:14.515832 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.515755    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"kube-root-ca.crt\""
Sep 10 12:51:14.517018 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.516070    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"openshift-service-ca.crt\""
Sep 10 12:51:14.521218 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.521193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"]
Sep 10 12:51:14.591221 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.591190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-dshm\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.591349 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.591231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-model-cache\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.591349 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.591264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-home\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.591349 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.591305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3add0af-726e-410b-91fc-f2d22ba198de-tls-certs\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.591462 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.591354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhj5q\" (UniqueName: \"kubernetes.io/projected/f3add0af-726e-410b-91fc-f2d22ba198de-kube-api-access-rhj5q\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.591462 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.591390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-tmp-dir\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.614230 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.614204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:14.692559 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.692526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3add0af-726e-410b-91fc-f2d22ba198de-tls-certs\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.692973 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.692585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhj5q\" (UniqueName: \"kubernetes.io/projected/f3add0af-726e-410b-91fc-f2d22ba198de-kube-api-access-rhj5q\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.692973 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.692657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-tmp-dir\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.692973 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.692718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-dshm\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.692973 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.692757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-model-cache\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.692973 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.692793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-home\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.693271 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.693247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-tmp-dir\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.693630 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.693578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-model-cache\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.693865 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.693585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-home\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.696176 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.696147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-dshm\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.696783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.696742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3add0af-726e-410b-91fc-f2d22ba198de-tls-certs\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.701742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.701722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhj5q\" (UniqueName: \"kubernetes.io/projected/f3add0af-726e-410b-91fc-f2d22ba198de-kube-api-access-rhj5q\") pod \"distinct-2-w0-30919a-kserve-76ff997f75-d2xdm\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") " pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:14.826192 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:14.826141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:15.099426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.099401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"]
Sep 10 12:51:15.101069 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:15.101043    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3add0af_726e_410b_91fc_f2d22ba198de.slice/crio-a3a8c43ca460131b00bf26b1fe5a499d135eccde5273cfe317871cf2b08940bf WatchSource:0}: Error finding container a3a8c43ca460131b00bf26b1fe5a499d135eccde5273cfe317871cf2b08940bf: Status 404 returned error can't find the container with id a3a8c43ca460131b00bf26b1fe5a499d135eccde5273cfe317871cf2b08940bf
Sep 10 12:51:15.531775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.531743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm" event={"ID":"f3add0af-726e-410b-91fc-f2d22ba198de","Type":"ContainerStarted","Data":"aa36b6a717943e21e63af9dff3e9e1887affd1b138e9b7110fe99d33b4867cda"}
Sep 10 12:51:15.531919 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.531786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm" event={"ID":"f3add0af-726e-410b-91fc-f2d22ba198de","Type":"ContainerStarted","Data":"a3a8c43ca460131b00bf26b1fe5a499d135eccde5273cfe317871cf2b08940bf"}
Sep 10 12:51:15.549455 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.549405    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm" podStartSLOduration=1.54938962 podStartE2EDuration="1.54938962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:15.548068145 +0000 UTC m=+1250.070237219" watchObservedRunningTime="2026-09-10 12:51:15.54938962 +0000 UTC m=+1250.071558671"
Sep 10 12:51:15.570799 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.570773    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"]
Sep 10 12:51:15.574645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.574630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:15.577003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.576987    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c5adae-dockercfg-tkrwj\""
Sep 10 12:51:15.577250 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.577234    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c5adae\""
Sep 10 12:51:15.583933 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.583909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"]
Sep 10 12:51:15.603232 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.603210    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"]
Sep 10 12:51:15.606914 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.606895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:15.616165 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.616145    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"]
Sep 10 12:51:15.702348 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.702317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d84e5f-6754-4ff2-ae30-535be48da7d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh\" (UID: \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:15.702738 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.702385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03291b8e-838f-44d4-a9b8-909fd648737e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td\" (UID: \"03291b8e-838f-44d4-a9b8-909fd648737e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:15.702738 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.702488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c967\" (UniqueName: \"kubernetes.io/projected/03291b8e-838f-44d4-a9b8-909fd648737e-kube-api-access-4c967\") pod \"payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td\" (UID: \"03291b8e-838f-44d4-a9b8-909fd648737e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:15.702738 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.702533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7xd\" (UniqueName: \"kubernetes.io/projected/c6d84e5f-6754-4ff2-ae30-535be48da7d9-kube-api-access-vk7xd\") pod \"payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh\" (UID: \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:15.803242 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.803181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d84e5f-6754-4ff2-ae30-535be48da7d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh\" (UID: \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:15.803242 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.803235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03291b8e-838f-44d4-a9b8-909fd648737e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td\" (UID: \"03291b8e-838f-44d4-a9b8-909fd648737e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:15.803440 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.803300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c967\" (UniqueName: \"kubernetes.io/projected/03291b8e-838f-44d4-a9b8-909fd648737e-kube-api-access-4c967\") pod \"payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td\" (UID: \"03291b8e-838f-44d4-a9b8-909fd648737e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:15.803440 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.803335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7xd\" (UniqueName: \"kubernetes.io/projected/c6d84e5f-6754-4ff2-ae30-535be48da7d9-kube-api-access-vk7xd\") pod \"payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh\" (UID: \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:15.804062 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.804041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d84e5f-6754-4ff2-ae30-535be48da7d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh\" (UID: \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:15.804147 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.804065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03291b8e-838f-44d4-a9b8-909fd648737e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td\" (UID: \"03291b8e-838f-44d4-a9b8-909fd648737e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:15.812203 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.812179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c967\" (UniqueName: \"kubernetes.io/projected/03291b8e-838f-44d4-a9b8-909fd648737e-kube-api-access-4c967\") pod \"payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td\" (UID: \"03291b8e-838f-44d4-a9b8-909fd648737e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:15.812442 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.812425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7xd\" (UniqueName: \"kubernetes.io/projected/c6d84e5f-6754-4ff2-ae30-535be48da7d9-kube-api-access-vk7xd\") pod \"payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh\" (UID: \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:15.886589 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.886553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:15.918028 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:15.917998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:16.178019 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.177992    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"]
Sep 10 12:51:16.179889 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:16.179857    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03291b8e_838f_44d4_a9b8_909fd648737e.slice/crio-aefd2b088929d5e22fb9b53cb8924c50e79886c6b8824e93e77eae2d0ce06e3e WatchSource:0}: Error finding container aefd2b088929d5e22fb9b53cb8924c50e79886c6b8824e93e77eae2d0ce06e3e: Status 404 returned error can't find the container with id aefd2b088929d5e22fb9b53cb8924c50e79886c6b8824e93e77eae2d0ce06e3e
Sep 10 12:51:16.217554 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.217526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"]
Sep 10 12:51:16.219728 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:16.219702    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d84e5f_6754_4ff2_ae30_535be48da7d9.slice/crio-75a30b43e30f3344b960df8a46d9647225d583c553400b2dedd99a8cb520599e WatchSource:0}: Error finding container 75a30b43e30f3344b960df8a46d9647225d583c553400b2dedd99a8cb520599e: Status 404 returned error can't find the container with id 75a30b43e30f3344b960df8a46d9647225d583c553400b2dedd99a8cb520599e
Sep 10 12:51:16.540957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.540921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td" event={"ID":"03291b8e-838f-44d4-a9b8-909fd648737e","Type":"ContainerStarted","Data":"b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d"}
Sep 10 12:51:16.541123 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.540998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:16.541123 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.541022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td" event={"ID":"03291b8e-838f-44d4-a9b8-909fd648737e","Type":"ContainerStarted","Data":"aefd2b088929d5e22fb9b53cb8924c50e79886c6b8824e93e77eae2d0ce06e3e"}
Sep 10 12:51:16.542769 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.542738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh" event={"ID":"c6d84e5f-6754-4ff2-ae30-535be48da7d9","Type":"ContainerStarted","Data":"5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9"}
Sep 10 12:51:16.542896 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.542790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh" event={"ID":"c6d84e5f-6754-4ff2-ae30-535be48da7d9","Type":"ContainerStarted","Data":"75a30b43e30f3344b960df8a46d9647225d583c553400b2dedd99a8cb520599e"}
Sep 10 12:51:16.542959 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.542936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:16.560128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.560084    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td" podStartSLOduration=1.560069029 podStartE2EDuration="1.560069029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:16.559131591 +0000 UTC m=+1251.081300649" watchObservedRunningTime="2026-09-10 12:51:16.560069029 +0000 UTC m=+1251.082238078"
Sep 10 12:51:16.578869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.578820    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh" podStartSLOduration=1.578806258 podStartE2EDuration="1.578806258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:16.577687312 +0000 UTC m=+1251.099856366" watchObservedRunningTime="2026-09-10 12:51:16.578806258 +0000 UTC m=+1251.100975309"
Sep 10 12:51:16.619480 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.619450    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"]
Sep 10 12:51:16.623522 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.623505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:16.626035 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.626005    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fce322fe-dockercfg-z4x44\""
Sep 10 12:51:16.626139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.626013    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fce322fe\""
Sep 10 12:51:16.638515 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.638492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"]
Sep 10 12:51:16.668922 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.668896    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"]
Sep 10 12:51:16.672485 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.672468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:16.686301 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.686272    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"]
Sep 10 12:51:16.713243 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.713218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38a4e05-e0ff-4a1b-af56-7de7764f7528-plugins-config-volume\") pod \"payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6\" (UID: \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\") " pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:16.713655 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.713310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47pc\" (UniqueName: \"kubernetes.io/projected/f38a4e05-e0ff-4a1b-af56-7de7764f7528-kube-api-access-r47pc\") pod \"payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6\" (UID: \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\") " pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:16.823442 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.823309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38a4e05-e0ff-4a1b-af56-7de7764f7528-plugins-config-volume\") pod \"payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6\" (UID: \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\") " pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:16.823813 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.823778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47pc\" (UniqueName: \"kubernetes.io/projected/f38a4e05-e0ff-4a1b-af56-7de7764f7528-kube-api-access-r47pc\") pod \"payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6\" (UID: \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\") " pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:16.823889 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.823871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3595da76-e8c3-4e6c-9c07-c907e52fcfed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp\" (UID: \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:16.824061 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.824044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57hpc\" (UniqueName: \"kubernetes.io/projected/3595da76-e8c3-4e6c-9c07-c907e52fcfed-kube-api-access-57hpc\") pod \"payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp\" (UID: \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:16.824559 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.824534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38a4e05-e0ff-4a1b-af56-7de7764f7528-plugins-config-volume\") pod \"payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6\" (UID: \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\") " pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:16.832890 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.832868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47pc\" (UniqueName: \"kubernetes.io/projected/f38a4e05-e0ff-4a1b-af56-7de7764f7528-kube-api-access-r47pc\") pod \"payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6\" (UID: \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\") " pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:16.925424 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.925388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3595da76-e8c3-4e6c-9c07-c907e52fcfed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp\" (UID: \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:16.925588 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.925445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hpc\" (UniqueName: \"kubernetes.io/projected/3595da76-e8c3-4e6c-9c07-c907e52fcfed-kube-api-access-57hpc\") pod \"payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp\" (UID: \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:16.926079 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.926060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3595da76-e8c3-4e6c-9c07-c907e52fcfed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp\" (UID: \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:16.934289 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.934262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:16.936486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.936456    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hpc\" (UniqueName: \"kubernetes.io/projected/3595da76-e8c3-4e6c-9c07-c907e52fcfed-kube-api-access-57hpc\") pod \"payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp\" (UID: \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:16.983222 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:16.983181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:17.228467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.228437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"]
Sep 10 12:51:17.229929 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:17.229901    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38a4e05_e0ff_4a1b_af56_7de7764f7528.slice/crio-3a3591521ce25237e0f345df5a41778a46311d9e372cc4825a439a3afc7e6e2b WatchSource:0}: Error finding container 3a3591521ce25237e0f345df5a41778a46311d9e372cc4825a439a3afc7e6e2b: Status 404 returned error can't find the container with id 3a3591521ce25237e0f345df5a41778a46311d9e372cc4825a439a3afc7e6e2b
Sep 10 12:51:17.266479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.266446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:17.287993 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.287880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"]
Sep 10 12:51:17.289682 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:17.289657    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3595da76_e8c3_4e6c_9c07_c907e52fcfed.slice/crio-971ab04b453c268eec3b189c8429a23c05012ff81abda613a438a2c5efe29bd7 WatchSource:0}: Error finding container 971ab04b453c268eec3b189c8429a23c05012ff81abda613a438a2c5efe29bd7: Status 404 returned error can't find the container with id 971ab04b453c268eec3b189c8429a23c05012ff81abda613a438a2c5efe29bd7
Sep 10 12:51:17.550198 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.550165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp" event={"ID":"3595da76-e8c3-4e6c-9c07-c907e52fcfed","Type":"ContainerStarted","Data":"aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5"}
Sep 10 12:51:17.550391 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.550228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp" event={"ID":"3595da76-e8c3-4e6c-9c07-c907e52fcfed","Type":"ContainerStarted","Data":"971ab04b453c268eec3b189c8429a23c05012ff81abda613a438a2c5efe29bd7"}
Sep 10 12:51:17.550450 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.550425    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:17.551975 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.551947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6" event={"ID":"f38a4e05-e0ff-4a1b-af56-7de7764f7528","Type":"ContainerStarted","Data":"082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3"}
Sep 10 12:51:17.552140 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.551992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6" event={"ID":"f38a4e05-e0ff-4a1b-af56-7de7764f7528","Type":"ContainerStarted","Data":"3a3591521ce25237e0f345df5a41778a46311d9e372cc4825a439a3afc7e6e2b"}
Sep 10 12:51:17.552140 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.552118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:17.572581 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.572541    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp" podStartSLOduration=1.572528428 podStartE2EDuration="1.572528428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:17.572103363 +0000 UTC m=+1252.094272415" watchObservedRunningTime="2026-09-10 12:51:17.572528428 +0000 UTC m=+1252.094697480"
Sep 10 12:51:17.600160 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.600122    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6" podStartSLOduration=1.600110152 podStartE2EDuration="1.600110152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:17.598934974 +0000 UTC m=+1252.121104043" watchObservedRunningTime="2026-09-10 12:51:17.600110152 +0000 UTC m=+1252.122279204"
Sep 10 12:51:17.986329 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:17.986212    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:20.131796 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:20.131758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:24.826971 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:24.826919    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:24.826971 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:24.826977    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:24.839699 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:24.839673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:25.621945 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:25.621917    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:51:26.086682 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:26.086652    2577 scope.go:117] "RemoveContainer" containerID="ba4159a16f7a216d64682076c77b21ef0deebe6fcc0bd951a2ac2472f2e726b8"
Sep 10 12:51:26.095157 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:26.095133    2577 scope.go:117] "RemoveContainer" containerID="74b31bb1cbbf31cbe418135d984234ae2e3e7e8e4793017c20207e0d7928a935"
Sep 10 12:51:27.554019 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:27.553982    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 12:51:27.554475 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:27.554051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 12:51:28.562257 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:28.562226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:51:28.562731 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:28.562301    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:51:35.992377 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:35.992344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:37.023651 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:37.023589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:38.308923 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:38.308881    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:38.591544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:38.591456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:38.681612 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:38.681569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:38.906139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:38.906043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:39.235058 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:39.234942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:39.601929 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:39.601892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:39.702053 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:39.702021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:42.672109 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:42.672075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:42.775290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:42.775256    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:43.877737 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:43.877703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:43.977017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:43.976987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:44.145146 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.145050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:44.489290 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.489193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:44.590118 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.590087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:51:44.712420 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.712389    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"]
Sep 10 12:51:44.718510 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.718489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:44.721637 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.721612    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e5208b-dockercfg-rr2hr\""
Sep 10 12:51:44.721747 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.721613    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e5208b\""
Sep 10 12:51:44.731439 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.731418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"]
Sep 10 12:51:44.777619 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.777583    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"]
Sep 10 12:51:44.781355 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.781339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:44.790448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.790424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccae0351-6797-4245-87a0-f04c89ab6aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm\" (UID: \"ccae0351-6797-4245-87a0-f04c89ab6aae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:44.790556 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.790483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqdv\" (UniqueName: \"kubernetes.io/projected/ccae0351-6797-4245-87a0-f04c89ab6aae-kube-api-access-cbqdv\") pod \"payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm\" (UID: \"ccae0351-6797-4245-87a0-f04c89ab6aae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:44.794233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.794211    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"]
Sep 10 12:51:44.891465 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.891433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0dc5dd-69c5-430a-ba61-a53795548a74-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm\" (UID: \"7f0dc5dd-69c5-430a-ba61-a53795548a74\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:44.891914 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.891532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cwv\" (UniqueName: \"kubernetes.io/projected/7f0dc5dd-69c5-430a-ba61-a53795548a74-kube-api-access-p4cwv\") pod \"payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm\" (UID: \"7f0dc5dd-69c5-430a-ba61-a53795548a74\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:44.891914 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.891618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccae0351-6797-4245-87a0-f04c89ab6aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm\" (UID: \"ccae0351-6797-4245-87a0-f04c89ab6aae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:44.891914 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.891662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqdv\" (UniqueName: \"kubernetes.io/projected/ccae0351-6797-4245-87a0-f04c89ab6aae-kube-api-access-cbqdv\") pod \"payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm\" (UID: \"ccae0351-6797-4245-87a0-f04c89ab6aae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:44.892432 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.892406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccae0351-6797-4245-87a0-f04c89ab6aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm\" (UID: \"ccae0351-6797-4245-87a0-f04c89ab6aae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:44.902029 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.902004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqdv\" (UniqueName: \"kubernetes.io/projected/ccae0351-6797-4245-87a0-f04c89ab6aae-kube-api-access-cbqdv\") pod \"payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm\" (UID: \"ccae0351-6797-4245-87a0-f04c89ab6aae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:44.993143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.993110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0dc5dd-69c5-430a-ba61-a53795548a74-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm\" (UID: \"7f0dc5dd-69c5-430a-ba61-a53795548a74\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:44.993269 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.993247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cwv\" (UniqueName: \"kubernetes.io/projected/7f0dc5dd-69c5-430a-ba61-a53795548a74-kube-api-access-p4cwv\") pod \"payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm\" (UID: \"7f0dc5dd-69c5-430a-ba61-a53795548a74\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:44.993767 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:44.993746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0dc5dd-69c5-430a-ba61-a53795548a74-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm\" (UID: \"7f0dc5dd-69c5-430a-ba61-a53795548a74\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:45.026021 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.025987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cwv\" (UniqueName: \"kubernetes.io/projected/7f0dc5dd-69c5-430a-ba61-a53795548a74-kube-api-access-p4cwv\") pod \"payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm\" (UID: \"7f0dc5dd-69c5-430a-ba61-a53795548a74\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:45.028688 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.028639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:45.091003 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.090904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:45.311645 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.311623    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"]
Sep 10 12:51:45.313661 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:45.313629    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccae0351_6797_4245_87a0_f04c89ab6aae.slice/crio-06e2f24f939f231caca43b033ac939bc6f090a3023bc8f76ac0b0135c86a1755 WatchSource:0}: Error finding container 06e2f24f939f231caca43b033ac939bc6f090a3023bc8f76ac0b0135c86a1755: Status 404 returned error can't find the container with id 06e2f24f939f231caca43b033ac939bc6f090a3023bc8f76ac0b0135c86a1755
Sep 10 12:51:45.376115 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:51:45.376091    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0dc5dd_69c5_430a_ba61_a53795548a74.slice/crio-a221b27652d4188c41e53588d08028d34b6eba44b308541f5133fb3a74c9f73f WatchSource:0}: Error finding container a221b27652d4188c41e53588d08028d34b6eba44b308541f5133fb3a74c9f73f: Status 404 returned error can't find the container with id a221b27652d4188c41e53588d08028d34b6eba44b308541f5133fb3a74c9f73f
Sep 10 12:51:45.376287 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.376265    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"]
Sep 10 12:51:45.725642 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.725533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm" event={"ID":"7f0dc5dd-69c5-430a-ba61-a53795548a74","Type":"ContainerStarted","Data":"6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283"}
Sep 10 12:51:45.725816 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.725724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:51:45.725816 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.725770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm" event={"ID":"7f0dc5dd-69c5-430a-ba61-a53795548a74","Type":"ContainerStarted","Data":"a221b27652d4188c41e53588d08028d34b6eba44b308541f5133fb3a74c9f73f"}
Sep 10 12:51:45.727110 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.727089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm" event={"ID":"ccae0351-6797-4245-87a0-f04c89ab6aae","Type":"ContainerStarted","Data":"fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999"}
Sep 10 12:51:45.727212 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.727126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm" event={"ID":"ccae0351-6797-4245-87a0-f04c89ab6aae","Type":"ContainerStarted","Data":"06e2f24f939f231caca43b033ac939bc6f090a3023bc8f76ac0b0135c86a1755"}
Sep 10 12:51:45.727212 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.727170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:45.762570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.762527    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm" podStartSLOduration=1.762514087 podStartE2EDuration="1.762514087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:45.760440776 +0000 UTC m=+1280.282609857" watchObservedRunningTime="2026-09-10 12:51:45.762514087 +0000 UTC m=+1280.284683138"
Sep 10 12:51:45.796114 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:45.796064    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm" podStartSLOduration=1.796046736 podStartE2EDuration="1.796046736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:45.793286951 +0000 UTC m=+1280.315456003" watchObservedRunningTime="2026-09-10 12:51:45.796046736 +0000 UTC m=+1280.318215793"
Sep 10 12:51:56.735218 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:56.735185    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 12:51:56.735625 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:51:56.735253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 12:52:07.707027 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:07.706991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:52:26.155296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:26.155263    2577 scope.go:117] "RemoveContainer" containerID="ae34edf81982e507ff96a255a87a21ccbc5b7f39f68f467551e6242c4f775fef"
Sep 10 12:52:27.641071 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:27.641044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:52:36.778154 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:36.778119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:52:38.073486 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:38.073455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:52:43.400127 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:43.400093    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:52:55.846343 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.846264    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"]
Sep 10 12:52:55.850198 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.850177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:55.861174 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.861148    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c5adae\"/\"kube-root-ca.crt\""
Sep 10 12:52:55.863822 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.863787    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c5adae\"/\"openshift-service-ca.crt\""
Sep 10 12:52:55.864046 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.863932    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c5adae\"/\"auth-test-model-c5adae-kserve-self-signed-certs\""
Sep 10 12:52:55.864046 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.863983    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c5adae\"/\"default-dockercfg-gzt7q\""
Sep 10 12:52:55.864535 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.864514    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"]
Sep 10 12:52:55.931980 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.931947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4825c6d9-ce15-426d-a079-a0aa6b804823-tls-certs\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:55.932150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.932007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-home\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:55.932150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.932025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-tmp-dir\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:55.932150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.932046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-dshm\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:55.932257 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.932140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-model-cache\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:55.932257 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.932219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxbb\" (UniqueName: \"kubernetes.io/projected/4825c6d9-ce15-426d-a079-a0aa6b804823-kube-api-access-cqxbb\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:55.982071 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:55.982039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:52:56.033076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-home\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.033076 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-tmp-dir\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.033322 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-dshm\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.033322 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-model-cache\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.033322 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxbb\" (UniqueName: \"kubernetes.io/projected/4825c6d9-ce15-426d-a079-a0aa6b804823-kube-api-access-cqxbb\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.033322 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4825c6d9-ce15-426d-a079-a0aa6b804823-tls-certs\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.033562 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-home\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.033660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-tmp-dir\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.033735 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.033681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-model-cache\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.035498 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.035466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-dshm\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.035639 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.035620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4825c6d9-ce15-426d-a079-a0aa6b804823-tls-certs\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.057679 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.057656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxbb\" (UniqueName: \"kubernetes.io/projected/4825c6d9-ce15-426d-a079-a0aa6b804823-kube-api-access-cqxbb\") pod \"auth-test-model-c5adae-kserve-7d84d5594c-lvccv\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.162327 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.162250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:52:56.451505 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:56.451481    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"]
Sep 10 12:52:56.453128 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:52:56.453103    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4825c6d9_ce15_426d_a079_a0aa6b804823.slice/crio-7393f5d49af0948fe1d4219e1d1161330cc94969c1869a883ffa5222fb5bfff1 WatchSource:0}: Error finding container 7393f5d49af0948fe1d4219e1d1161330cc94969c1869a883ffa5222fb5bfff1: Status 404 returned error can't find the container with id 7393f5d49af0948fe1d4219e1d1161330cc94969c1869a883ffa5222fb5bfff1
Sep 10 12:52:57.123994 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:57.123957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv" event={"ID":"4825c6d9-ce15-426d-a079-a0aa6b804823","Type":"ContainerStarted","Data":"c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721"}
Sep 10 12:52:57.124359 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:52:57.124006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv" event={"ID":"4825c6d9-ce15-426d-a079-a0aa6b804823","Type":"ContainerStarted","Data":"7393f5d49af0948fe1d4219e1d1161330cc94969c1869a883ffa5222fb5bfff1"}
Sep 10 12:53:02.836669 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:02.836632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:53:06.163347 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:06.163315    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:53:06.163761 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:06.163382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:53:06.176426 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:06.176402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:53:06.204127 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:06.204084    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv" podStartSLOduration=11.204070311 podStartE2EDuration="11.204070311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:52:57.143571087 +0000 UTC m=+1351.665740140" watchObservedRunningTime="2026-09-10 12:53:06.204070311 +0000 UTC m=+1360.726239362"
Sep 10 12:53:07.190005 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:07.189976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 12:53:13.651433 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:13.651308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:53:13.749446 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:13.749399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:53:25.889359 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:25.889327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:53:46.235362 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.235329    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"]
Sep 10 12:53:46.238746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.238724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"
Sep 10 12:53:46.244658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.244635    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:53:46.270173 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.270147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"]
Sep 10 12:53:46.304767 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.304735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de103591-af34-450e-8c8e-af2daf9d0fbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fce322fe-xcfft\" (UID: \"de103591-af34-450e-8c8e-af2daf9d0fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"
Sep 10 12:53:46.406015 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.405982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de103591-af34-450e-8c8e-af2daf9d0fbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fce322fe-xcfft\" (UID: \"de103591-af34-450e-8c8e-af2daf9d0fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"
Sep 10 12:53:46.406652 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.406635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de103591-af34-450e-8c8e-af2daf9d0fbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fce322fe-xcfft\" (UID: \"de103591-af34-450e-8c8e-af2daf9d0fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"
Sep 10 12:53:46.430120 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.430097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:53:46.548284 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.548261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"
Sep 10 12:53:46.830436 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:46.830404    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"]
Sep 10 12:53:46.831927 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:53:46.831899    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde103591_af34_450e_8c8e_af2daf9d0fbd.slice/crio-e49b40c669180b8b5e0ebe240671f8cec020d6e48b1e765a4f51af1071fc0756 WatchSource:0}: Error finding container e49b40c669180b8b5e0ebe240671f8cec020d6e48b1e765a4f51af1071fc0756: Status 404 returned error can't find the container with id e49b40c669180b8b5e0ebe240671f8cec020d6e48b1e765a4f51af1071fc0756
Sep 10 12:53:47.400213 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:47.400099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft" event={"ID":"de103591-af34-450e-8c8e-af2daf9d0fbd","Type":"ContainerStarted","Data":"71f23648012d6daace2ba21a6e5298b3f27eeb1e66bdea49b91134a4dc733b37"}
Sep 10 12:53:47.400563 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:47.400276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft" event={"ID":"de103591-af34-450e-8c8e-af2daf9d0fbd","Type":"ContainerStarted","Data":"e49b40c669180b8b5e0ebe240671f8cec020d6e48b1e765a4f51af1071fc0756"}
Sep 10 12:53:47.447513 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:47.447466    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft" podStartSLOduration=1.447450812 podStartE2EDuration="1.447450812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:53:47.442729095 +0000 UTC m=+1401.964898147" watchObservedRunningTime="2026-09-10 12:53:47.447450812 +0000 UTC m=+1401.969619864"
Sep 10 12:53:48.408114 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:48.408083    2577 generic.go:356] "Generic (PLEG): container finished" podID="de103591-af34-450e-8c8e-af2daf9d0fbd" containerID="71f23648012d6daace2ba21a6e5298b3f27eeb1e66bdea49b91134a4dc733b37" exitCode=0
Sep 10 12:53:48.408488 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:48.408167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft" event={"ID":"de103591-af34-450e-8c8e-af2daf9d0fbd","Type":"ContainerDied","Data":"71f23648012d6daace2ba21a6e5298b3f27eeb1e66bdea49b91134a4dc733b37"}
Sep 10 12:53:49.556024 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:49.556001    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"
Sep 10 12:53:49.640253 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:49.640225    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de103591-af34-450e-8c8e-af2daf9d0fbd-maas-api-service-ca\") pod \"de103591-af34-450e-8c8e-af2daf9d0fbd\" (UID: \"de103591-af34-450e-8c8e-af2daf9d0fbd\") "
Sep 10 12:53:49.640563 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:49.640540    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de103591-af34-450e-8c8e-af2daf9d0fbd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de103591-af34-450e-8c8e-af2daf9d0fbd" (UID: "de103591-af34-450e-8c8e-af2daf9d0fbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:53:49.741042 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:49.740977    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de103591-af34-450e-8c8e-af2daf9d0fbd-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:53:50.420390 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:50.420364    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"
Sep 10 12:53:50.420563 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:50.420404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft" event={"ID":"de103591-af34-450e-8c8e-af2daf9d0fbd","Type":"ContainerDied","Data":"e49b40c669180b8b5e0ebe240671f8cec020d6e48b1e765a4f51af1071fc0756"}
Sep 10 12:53:50.420563 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:53:50.420453    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e49b40c669180b8b5e0ebe240671f8cec020d6e48b1e765a4f51af1071fc0756"
Sep 10 12:54:02.359273 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.359169    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:02.397306 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.397268    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"]
Sep 10 12:54:02.397532 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.397486    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6" podUID="f38a4e05-e0ff-4a1b-af56-7de7764f7528" containerName="payload-processing" containerID="cri-o://082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3" gracePeriod=30
Sep 10 12:54:02.413608 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.413561    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"]
Sep 10 12:54:02.414039 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.414008    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp" podUID="3595da76-e8c3-4e6c-9c07-c907e52fcfed" containerName="payload-pre-processing" containerID="cri-o://aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5" gracePeriod=30
Sep 10 12:54:02.462406 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.462370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:02.571899 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.571864    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:02.656117 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.656086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:02.691454 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.691419    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"]
Sep 10 12:54:02.700624 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.700338    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fce322fe-xcfft"]
Sep 10 12:54:02.707583 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.707555    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:54:02.709783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.709760    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:54:02.751847 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.751817    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:02.762415 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.762386    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38a4e05-e0ff-4a1b-af56-7de7764f7528-plugins-config-volume\") pod \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\" (UID: \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\") "
Sep 10 12:54:02.762415 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.762421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57hpc\" (UniqueName: \"kubernetes.io/projected/3595da76-e8c3-4e6c-9c07-c907e52fcfed-kube-api-access-57hpc\") pod \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\" (UID: \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\") "
Sep 10 12:54:02.762660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.762466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3595da76-e8c3-4e6c-9c07-c907e52fcfed-plugins-config-volume\") pod \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\" (UID: \"3595da76-e8c3-4e6c-9c07-c907e52fcfed\") "
Sep 10 12:54:02.762660 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.762518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r47pc\" (UniqueName: \"kubernetes.io/projected/f38a4e05-e0ff-4a1b-af56-7de7764f7528-kube-api-access-r47pc\") pod \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\" (UID: \"f38a4e05-e0ff-4a1b-af56-7de7764f7528\") "
Sep 10 12:54:02.762938 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.762889    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3595da76-e8c3-4e6c-9c07-c907e52fcfed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3595da76-e8c3-4e6c-9c07-c907e52fcfed" (UID: "3595da76-e8c3-4e6c-9c07-c907e52fcfed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:54:02.762938 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.762919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38a4e05-e0ff-4a1b-af56-7de7764f7528-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f38a4e05-e0ff-4a1b-af56-7de7764f7528" (UID: "f38a4e05-e0ff-4a1b-af56-7de7764f7528"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:54:02.765736 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.765688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38a4e05-e0ff-4a1b-af56-7de7764f7528-kube-api-access-r47pc" (OuterVolumeSpecName: "kube-api-access-r47pc") pod "f38a4e05-e0ff-4a1b-af56-7de7764f7528" (UID: "f38a4e05-e0ff-4a1b-af56-7de7764f7528"). InnerVolumeSpecName "kube-api-access-r47pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:54:02.766316 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.765712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3595da76-e8c3-4e6c-9c07-c907e52fcfed-kube-api-access-57hpc" (OuterVolumeSpecName: "kube-api-access-57hpc") pod "3595da76-e8c3-4e6c-9c07-c907e52fcfed" (UID: "3595da76-e8c3-4e6c-9c07-c907e52fcfed"). InnerVolumeSpecName "kube-api-access-57hpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:54:02.864017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.863976    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r47pc\" (UniqueName: \"kubernetes.io/projected/f38a4e05-e0ff-4a1b-af56-7de7764f7528-kube-api-access-r47pc\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:54:02.864017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.864010    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f38a4e05-e0ff-4a1b-af56-7de7764f7528-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:54:02.864017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.864024    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57hpc\" (UniqueName: \"kubernetes.io/projected/3595da76-e8c3-4e6c-9c07-c907e52fcfed-kube-api-access-57hpc\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:54:02.864258 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:02.864035    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3595da76-e8c3-4e6c-9c07-c907e52fcfed-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:54:03.493913 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.493877    2577 generic.go:356] "Generic (PLEG): container finished" podID="3595da76-e8c3-4e6c-9c07-c907e52fcfed" containerID="aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5" exitCode=0
Sep 10 12:54:03.494342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.493927    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"
Sep 10 12:54:03.494342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.493967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp" event={"ID":"3595da76-e8c3-4e6c-9c07-c907e52fcfed","Type":"ContainerDied","Data":"aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5"}
Sep 10 12:54:03.494342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.494023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp" event={"ID":"3595da76-e8c3-4e6c-9c07-c907e52fcfed","Type":"ContainerDied","Data":"971ab04b453c268eec3b189c8429a23c05012ff81abda613a438a2c5efe29bd7"}
Sep 10 12:54:03.494342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.494044    2577 scope.go:117] "RemoveContainer" containerID="aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5"
Sep 10 12:54:03.495470 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.495453    2577 generic.go:356] "Generic (PLEG): container finished" podID="f38a4e05-e0ff-4a1b-af56-7de7764f7528" containerID="082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3" exitCode=0
Sep 10 12:54:03.495554 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.495488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6" event={"ID":"f38a4e05-e0ff-4a1b-af56-7de7764f7528","Type":"ContainerDied","Data":"082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3"}
Sep 10 12:54:03.495554 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.495506    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"
Sep 10 12:54:03.495554 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.495525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6" event={"ID":"f38a4e05-e0ff-4a1b-af56-7de7764f7528","Type":"ContainerDied","Data":"3a3591521ce25237e0f345df5a41778a46311d9e372cc4825a439a3afc7e6e2b"}
Sep 10 12:54:03.503819 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.503662    2577 scope.go:117] "RemoveContainer" containerID="aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5"
Sep 10 12:54:03.504105 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:54:03.504083    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5\": container with ID starting with aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5 not found: ID does not exist" containerID="aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5"
Sep 10 12:54:03.504202 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.504113    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5"} err="failed to get container status \"aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5\": rpc error: code = NotFound desc = could not find container \"aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5\": container with ID starting with aa7b6aa5d26cfd8e96f0e433f796f5b3e2b3b6514e62cd731a4bb4eaee6ac4e5 not found: ID does not exist"
Sep 10 12:54:03.504202 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.504137    2577 scope.go:117] "RemoveContainer" containerID="082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3"
Sep 10 12:54:03.512462 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.512446    2577 scope.go:117] "RemoveContainer" containerID="082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3"
Sep 10 12:54:03.512738 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:54:03.512716    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3\": container with ID starting with 082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3 not found: ID does not exist" containerID="082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3"
Sep 10 12:54:03.512799 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.512744    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3"} err="failed to get container status \"082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3\": rpc error: code = NotFound desc = could not find container \"082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3\": container with ID starting with 082ab9aaa16c5719d68c84f142d0979d5ac16f3b676bd789591fd085331573b3 not found: ID does not exist"
Sep 10 12:54:03.522462 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.522436    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"]
Sep 10 12:54:03.528268 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.528248    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fce322fe-c7cf5bf86-ttxpp"]
Sep 10 12:54:03.542507 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.542486    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"]
Sep 10 12:54:03.551992 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:03.551973    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fce322fe-6f59d4c569-97qg6"]
Sep 10 12:54:04.000670 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:04.000636    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3595da76-e8c3-4e6c-9c07-c907e52fcfed" path="/var/lib/kubelet/pods/3595da76-e8c3-4e6c-9c07-c907e52fcfed/volumes"
Sep 10 12:54:04.001024 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:04.001011    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de103591-af34-450e-8c8e-af2daf9d0fbd" path="/var/lib/kubelet/pods/de103591-af34-450e-8c8e-af2daf9d0fbd/volumes"
Sep 10 12:54:04.001328 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:04.001315    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38a4e05-e0ff-4a1b-af56-7de7764f7528" path="/var/lib/kubelet/pods/f38a4e05-e0ff-4a1b-af56-7de7764f7528/volumes"
Sep 10 12:54:06.299623 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:06.299572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:11.350300 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.350266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:11.743288 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743209    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"]
Sep 10 12:54:11.743713 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743699    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de103591-af34-450e-8c8e-af2daf9d0fbd" containerName="revoke-keys"
Sep 10 12:54:11.743772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743715    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de103591-af34-450e-8c8e-af2daf9d0fbd" containerName="revoke-keys"
Sep 10 12:54:11.743772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743724    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3595da76-e8c3-4e6c-9c07-c907e52fcfed" containerName="payload-pre-processing"
Sep 10 12:54:11.743772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743731    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3595da76-e8c3-4e6c-9c07-c907e52fcfed" containerName="payload-pre-processing"
Sep 10 12:54:11.743772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743752    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38a4e05-e0ff-4a1b-af56-7de7764f7528" containerName="payload-processing"
Sep 10 12:54:11.743772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743758    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38a4e05-e0ff-4a1b-af56-7de7764f7528" containerName="payload-processing"
Sep 10 12:54:11.743957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743844    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3595da76-e8c3-4e6c-9c07-c907e52fcfed" containerName="payload-pre-processing"
Sep 10 12:54:11.743957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743854    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38a4e05-e0ff-4a1b-af56-7de7764f7528" containerName="payload-processing"
Sep 10 12:54:11.743957 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.743861    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de103591-af34-450e-8c8e-af2daf9d0fbd" containerName="revoke-keys"
Sep 10 12:54:11.748673 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.748643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.753766 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.753744    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"kube-root-ca.crt\""
Sep 10 12:54:11.754749 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.754721    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"embedding-w4-1312ea-kserve-self-signed-certs\""
Sep 10 12:54:11.754866 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.754756    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"default-dockercfg-q5q96\""
Sep 10 12:54:11.757086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.757069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"openshift-service-ca.crt\""
Sep 10 12:54:11.767324 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.767300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"]
Sep 10 12:54:11.769633 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.769584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:11.846545 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.846513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-model-cache\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.846743 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.846650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-dshm\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.846743 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.846679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr42k\" (UniqueName: \"kubernetes.io/projected/763ee82a-1a3c-4bd9-8315-61e7fc00d191-kube-api-access-kr42k\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.846743 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.846712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-home\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.846862 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.846794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tls-certs\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.846862 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.846827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tmp-dir\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.947979 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.947947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-model-cache\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.948142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.948022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-dshm\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.948142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.948040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr42k\" (UniqueName: \"kubernetes.io/projected/763ee82a-1a3c-4bd9-8315-61e7fc00d191-kube-api-access-kr42k\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.948142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.948067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-home\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.948142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.948106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tls-certs\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.948142 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.948136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tmp-dir\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.948392 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.948357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-model-cache\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.948453 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.948435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-home\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.948511 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.948463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tmp-dir\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.950205 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.950184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-dshm\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.950641 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.950617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tls-certs\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:11.960128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:11.960106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr42k\" (UniqueName: \"kubernetes.io/projected/763ee82a-1a3c-4bd9-8315-61e7fc00d191-kube-api-access-kr42k\") pod \"embedding-w4-1312ea-kserve-575b45bc75-r5x48\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") " pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:12.058497 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:12.058471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:12.338477 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:12.338452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"]
Sep 10 12:54:12.340735 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:54:12.340703    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod763ee82a_1a3c_4bd9_8315_61e7fc00d191.slice/crio-6d8167e9eba866910eedcd63382e9b0bad52054bf0266c0843f8cbbde21dc174 WatchSource:0}: Error finding container 6d8167e9eba866910eedcd63382e9b0bad52054bf0266c0843f8cbbde21dc174: Status 404 returned error can't find the container with id 6d8167e9eba866910eedcd63382e9b0bad52054bf0266c0843f8cbbde21dc174
Sep 10 12:54:12.547656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:12.547614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48" event={"ID":"763ee82a-1a3c-4bd9-8315-61e7fc00d191","Type":"ContainerStarted","Data":"42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014"}
Sep 10 12:54:12.547656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:12.547660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48" event={"ID":"763ee82a-1a3c-4bd9-8315-61e7fc00d191","Type":"ContainerStarted","Data":"6d8167e9eba866910eedcd63382e9b0bad52054bf0266c0843f8cbbde21dc174"}
Sep 10 12:54:12.574017 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:12.573971    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48" podStartSLOduration=1.573951256 podStartE2EDuration="1.573951256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:54:12.572052072 +0000 UTC m=+1427.094221124" watchObservedRunningTime="2026-09-10 12:54:12.573951256 +0000 UTC m=+1427.096120309"
Sep 10 12:54:22.059489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:22.059392    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:22.059489 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:22.059448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:22.072081 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:22.072059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:22.615480 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:22.615449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:54:27.050966 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.050922    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"]
Sep 10 12:54:27.056151 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.056126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.064499 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.064182    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"openshift-service-ca.crt\""
Sep 10 12:54:27.064721 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.064697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"kube-root-ca.crt\""
Sep 10 12:54:27.064823 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.064763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"default-dockercfg-2lpg6\""
Sep 10 12:54:27.065099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.065079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"auth-test-model-e5208b-kserve-self-signed-certs\""
Sep 10 12:54:27.087431 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.087404    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"]
Sep 10 12:54:27.107067 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.107014    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:27.188383 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.188337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-dshm\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.188565 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.188399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-home\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.188565 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.188502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-model-cache\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.188713 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.188582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tls-certs\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.188713 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.188644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xqn\" (UniqueName: \"kubernetes.io/projected/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-kube-api-access-g9xqn\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.188713 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.188707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tmp-dir\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.289961 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.289926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-dshm\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.290139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.289984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-home\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.290139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.290058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-model-cache\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.290139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.290092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tls-certs\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.290139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.290121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xqn\" (UniqueName: \"kubernetes.io/projected/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-kube-api-access-g9xqn\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.290364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.290183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tmp-dir\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.290512 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.290488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-model-cache\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.290609 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.290538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-home\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.290674 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.290613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tmp-dir\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.292405 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.292381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-dshm\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.292577 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.292561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tls-certs\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.307314 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.307255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xqn\" (UniqueName: \"kubernetes.io/projected/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-kube-api-access-g9xqn\") pod \"auth-test-model-e5208b-kserve-7f5769c5ff-55hxj\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.367461 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.367427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:27.654430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:27.654401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"]
Sep 10 12:54:27.656829 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:54:27.656790    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f47cbc6_80d8_4e6a_a1e5_021c77454bbb.slice/crio-6a0ca12c7edfb0ae58ece9cae277b2e75aa4cf9dbedc07e4ee6d5ec1951b7bc3 WatchSource:0}: Error finding container 6a0ca12c7edfb0ae58ece9cae277b2e75aa4cf9dbedc07e4ee6d5ec1951b7bc3: Status 404 returned error can't find the container with id 6a0ca12c7edfb0ae58ece9cae277b2e75aa4cf9dbedc07e4ee6d5ec1951b7bc3
Sep 10 12:54:28.644424 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:28.644384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj" event={"ID":"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb","Type":"ContainerStarted","Data":"cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050"}
Sep 10 12:54:28.644903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:28.644435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj" event={"ID":"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb","Type":"ContainerStarted","Data":"6a0ca12c7edfb0ae58ece9cae277b2e75aa4cf9dbedc07e4ee6d5ec1951b7bc3"}
Sep 10 12:54:28.678929 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:28.678880    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj" podStartSLOduration=1.6788650870000001 podStartE2EDuration="1.678865087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:54:28.67649373 +0000 UTC m=+1443.198662782" watchObservedRunningTime="2026-09-10 12:54:28.678865087 +0000 UTC m=+1443.201034169"
Sep 10 12:54:32.502122 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:32.502092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:33.775116 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:33.775081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:37.368495 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:37.368456    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:37.368927 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:37.368519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:37.380975 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:37.380951    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:37.705737 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:37.705663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 12:54:46.574735 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:46.574688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:50.797570 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:50.797540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:54:51.433212 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:54:51.433158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:20.597460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:20.597427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:21.865294 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:21.865260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:26.010082 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:26.010054    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:55:26.015042 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:26.015019    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:55:26.015176 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:26.015027    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 12:55:26.023197 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:26.023178    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 12:55:29.614307 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:29.614258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:36.636437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:36.636403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:38.030752 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:38.030719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:38.143947 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:38.143918    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:38.261506 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:38.261470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:39.296028 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:39.295997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:39.459823 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:39.459787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:39.641820 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:39.641740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:41.279201 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:41.279171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:44.781356 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:44.781322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:44.880562 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:44.880522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:50.222026 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:50.221938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:50.365477 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:50.365442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:51.130557 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:51.130518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:52.943517 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:52.943459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:55:58.300470 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:55:58.300431    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:02.018529 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:02.018492    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:02.121286 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:02.121255    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:11.633808 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:11.633770    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:11.882263 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:11.882229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:32.896433 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:32.896393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:37.038868 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:37.038830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:40.023816 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:40.023777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:40.125313 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:40.125277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:40.433453 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:40.433368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:40.974802 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:40.974773    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:41.356897 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:41.356868    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:44.604199 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:44.604166    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:44.693099 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:44.693064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:46.091999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:46.091968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:46.212387 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:46.212360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:46.991687 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:46.991655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:47.114730 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.114696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:56:47.274544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.274511    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"]
Sep 10 12:56:47.278275 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.278259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:47.281293 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.281266    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fe8579\""
Sep 10 12:56:47.283455 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.283438    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fe8579-dockercfg-zzpbn\""
Sep 10 12:56:47.296167 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.296140    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"]
Sep 10 12:56:47.308518 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.308491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe8f292-3054-4bbd-84ba-e05a70a250b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4\" (UID: \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:47.308658 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.308557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czstj\" (UniqueName: \"kubernetes.io/projected/bbe8f292-3054-4bbd-84ba-e05a70a250b0-kube-api-access-czstj\") pod \"payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4\" (UID: \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:47.409483 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.409451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe8f292-3054-4bbd-84ba-e05a70a250b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4\" (UID: \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:47.409675 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.409526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czstj\" (UniqueName: \"kubernetes.io/projected/bbe8f292-3054-4bbd-84ba-e05a70a250b0-kube-api-access-czstj\") pod \"payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4\" (UID: \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:47.410218 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.410199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe8f292-3054-4bbd-84ba-e05a70a250b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4\" (UID: \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:47.431722 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.431693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czstj\" (UniqueName: \"kubernetes.io/projected/bbe8f292-3054-4bbd-84ba-e05a70a250b0-kube-api-access-czstj\") pod \"payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4\" (UID: \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:47.589127 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.589055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:47.882824 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:47.882799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"]
Sep 10 12:56:47.884947 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:56:47.884921    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe8f292_3054_4bbd_84ba_e05a70a250b0.slice/crio-7cd1a5695e27c31db74f7bf5d19e948a3ac81bef21e9f70b2b1a70062e0ee745 WatchSource:0}: Error finding container 7cd1a5695e27c31db74f7bf5d19e948a3ac81bef21e9f70b2b1a70062e0ee745: Status 404 returned error can't find the container with id 7cd1a5695e27c31db74f7bf5d19e948a3ac81bef21e9f70b2b1a70062e0ee745
Sep 10 12:56:48.440029 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:48.439989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4" event={"ID":"bbe8f292-3054-4bbd-84ba-e05a70a250b0","Type":"ContainerStarted","Data":"091713c3cbe380f1bba231b7c2b2fd884f924d29408ca474348c27127f9148a3"}
Sep 10 12:56:48.440416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:48.440054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4" event={"ID":"bbe8f292-3054-4bbd-84ba-e05a70a250b0","Type":"ContainerStarted","Data":"7cd1a5695e27c31db74f7bf5d19e948a3ac81bef21e9f70b2b1a70062e0ee745"}
Sep 10 12:56:48.440416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:48.440105    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:56:48.471139 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:48.471094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4" podStartSLOduration=1.471078925 podStartE2EDuration="1.471078925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:56:48.469462565 +0000 UTC m=+1582.991631616" watchObservedRunningTime="2026-09-10 12:56:48.471078925 +0000 UTC m=+1582.993247977"
Sep 10 12:56:59.447508 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:56:59.447476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:57:05.894904 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:05.894863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:07.894430 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:07.894398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:09.656713 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:09.656680    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:10.143845 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:10.143811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:10.530621 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:10.530580    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:10.790423 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:10.790330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:13.767572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:13.767526    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:13.889637 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:13.889582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:15.095933 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:15.095902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:15.216234 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:15.216192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:15.316624 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:15.316564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:16.090746 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.090716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:16.246495 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.246467    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:16.369745 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.369669    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"]
Sep 10 12:57:16.373442 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.373424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:16.380855 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.380835    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b3c07b\""
Sep 10 12:57:16.401812 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.401788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"]
Sep 10 12:57:16.485110 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.485079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56kbc\" (UniqueName: \"kubernetes.io/projected/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-kube-api-access-56kbc\") pod \"payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9\" (UID: \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:16.485240 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.485186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9\" (UID: \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:16.497639 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.497610    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"]
Sep 10 12:57:16.501255 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.501238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:16.513826 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.513807    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b3c07b-dockercfg-nkkgc\""
Sep 10 12:57:16.527396 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.527376    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"]
Sep 10 12:57:16.586105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.586074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9\" (UID: \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:16.586240 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.586163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56kbc\" (UniqueName: \"kubernetes.io/projected/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-kube-api-access-56kbc\") pod \"payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9\" (UID: \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:16.586838 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.586818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9\" (UID: \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:16.604070 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.604045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56kbc\" (UniqueName: \"kubernetes.io/projected/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-kube-api-access-56kbc\") pod \"payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9\" (UID: \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:16.684201 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.684143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:16.687356 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.687098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxzz\" (UniqueName: \"kubernetes.io/projected/e5747814-5c17-4bc2-a167-66d95a8a9de1-kube-api-access-9jxzz\") pod \"payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd\" (UID: \"e5747814-5c17-4bc2-a167-66d95a8a9de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:16.687356 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.687159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5747814-5c17-4bc2-a167-66d95a8a9de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd\" (UID: \"e5747814-5c17-4bc2-a167-66d95a8a9de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:16.787815 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.787777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxzz\" (UniqueName: \"kubernetes.io/projected/e5747814-5c17-4bc2-a167-66d95a8a9de1-kube-api-access-9jxzz\") pod \"payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd\" (UID: \"e5747814-5c17-4bc2-a167-66d95a8a9de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:16.787962 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.787861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5747814-5c17-4bc2-a167-66d95a8a9de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd\" (UID: \"e5747814-5c17-4bc2-a167-66d95a8a9de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:16.788831 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.788808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5747814-5c17-4bc2-a167-66d95a8a9de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd\" (UID: \"e5747814-5c17-4bc2-a167-66d95a8a9de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:16.805578 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.805546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxzz\" (UniqueName: \"kubernetes.io/projected/e5747814-5c17-4bc2-a167-66d95a8a9de1-kube-api-access-9jxzz\") pod \"payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd\" (UID: \"e5747814-5c17-4bc2-a167-66d95a8a9de1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:16.813506 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.813475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:16.987006 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:16.986979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"]
Sep 10 12:57:16.988719 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:57:16.988682    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f15272_9bb4_41f2_9e9c_d674ab5a2fc1.slice/crio-f0c55ce7f6d9da086816111157f9bcc3cff847d4609ad20bffb2bf8b2aa8d46b WatchSource:0}: Error finding container f0c55ce7f6d9da086816111157f9bcc3cff847d4609ad20bffb2bf8b2aa8d46b: Status 404 returned error can't find the container with id f0c55ce7f6d9da086816111157f9bcc3cff847d4609ad20bffb2bf8b2aa8d46b
Sep 10 12:57:17.126554 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.126513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"]
Sep 10 12:57:17.129674 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:57:17.129640    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5747814_5c17_4bc2_a167_66d95a8a9de1.slice/crio-0dbaafb9f3ea265ebccb783b2870e69eb747ce6f713f220e923d30202435c100 WatchSource:0}: Error finding container 0dbaafb9f3ea265ebccb783b2870e69eb747ce6f713f220e923d30202435c100: Status 404 returned error can't find the container with id 0dbaafb9f3ea265ebccb783b2870e69eb747ce6f713f220e923d30202435c100
Sep 10 12:57:17.617797 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.617749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd" event={"ID":"e5747814-5c17-4bc2-a167-66d95a8a9de1","Type":"ContainerStarted","Data":"a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8"}
Sep 10 12:57:17.618234 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.617814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd" event={"ID":"e5747814-5c17-4bc2-a167-66d95a8a9de1","Type":"ContainerStarted","Data":"0dbaafb9f3ea265ebccb783b2870e69eb747ce6f713f220e923d30202435c100"}
Sep 10 12:57:17.618234 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.617969    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:17.619460 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.619431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9" event={"ID":"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1","Type":"ContainerStarted","Data":"a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9"}
Sep 10 12:57:17.619617 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.619480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9" event={"ID":"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1","Type":"ContainerStarted","Data":"f0c55ce7f6d9da086816111157f9bcc3cff847d4609ad20bffb2bf8b2aa8d46b"}
Sep 10 12:57:17.619617 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.619506    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:17.690029 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.689987    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd" podStartSLOduration=1.6899734039999998 podStartE2EDuration="1.689973404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:57:17.688490506 +0000 UTC m=+1612.210659558" watchObservedRunningTime="2026-09-10 12:57:17.689973404 +0000 UTC m=+1612.212142455"
Sep 10 12:57:17.742759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:17.742707    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9" podStartSLOduration=1.742690464 podStartE2EDuration="1.742690464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:57:17.742223422 +0000 UTC m=+1612.264392493" watchObservedRunningTime="2026-09-10 12:57:17.742690464 +0000 UTC m=+1612.264859517"
Sep 10 12:57:26.224208 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:26.224176    2577 scope.go:117] "RemoveContainer" containerID="12a6926b3c46585ef8f78c7c5370d7d0146835c2916b3277acd724f909f44cb3"
Sep 10 12:57:28.627023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:28.626991    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:57:28.627431 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:28.627313    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:57:38.741661 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:38.741626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:41.865933 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:41.865893    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"]
Sep 10 12:57:41.870655 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:41.870630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"
Sep 10 12:57:41.874933 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:41.874909    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:57:41.886101 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:41.886019    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"]
Sep 10 12:57:41.936028 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:41.935997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e6d95a-83a3-44d5-9997-3fbbb01635a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z\" (UID: \"a7e6d95a-83a3-44d5-9997-3fbbb01635a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"
Sep 10 12:57:42.034654 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:42.034614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:42.036918 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:42.036892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e6d95a-83a3-44d5-9997-3fbbb01635a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z\" (UID: \"a7e6d95a-83a3-44d5-9997-3fbbb01635a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"
Sep 10 12:57:42.037498 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:42.037480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e6d95a-83a3-44d5-9997-3fbbb01635a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z\" (UID: \"a7e6d95a-83a3-44d5-9997-3fbbb01635a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"
Sep 10 12:57:42.182410 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:42.182320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"
Sep 10 12:57:42.459105 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:42.459077    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"]
Sep 10 12:57:42.462110 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:57:42.462075    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e6d95a_83a3_44d5_9997_3fbbb01635a2.slice/crio-b56fb90f2e829afdb5affe3afacd4c25b0b521e5bb953ce0b09d81ebced71b7e WatchSource:0}: Error finding container b56fb90f2e829afdb5affe3afacd4c25b0b521e5bb953ce0b09d81ebced71b7e: Status 404 returned error can't find the container with id b56fb90f2e829afdb5affe3afacd4c25b0b521e5bb953ce0b09d81ebced71b7e
Sep 10 12:57:42.766422 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:42.766392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z" event={"ID":"a7e6d95a-83a3-44d5-9997-3fbbb01635a2","Type":"ContainerStarted","Data":"e7e8d3c4eea4116f40f6b728eb8b59616914f25e479feb9d742d8a349627b823"}
Sep 10 12:57:42.766553 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:42.766442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z" event={"ID":"a7e6d95a-83a3-44d5-9997-3fbbb01635a2","Type":"ContainerStarted","Data":"b56fb90f2e829afdb5affe3afacd4c25b0b521e5bb953ce0b09d81ebced71b7e"}
Sep 10 12:57:42.789254 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:42.789210    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z" podStartSLOduration=1.789192265 podStartE2EDuration="1.789192265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:57:42.786480569 +0000 UTC m=+1637.308649620" watchObservedRunningTime="2026-09-10 12:57:42.789192265 +0000 UTC m=+1637.311361360"
Sep 10 12:57:43.773926 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:43.773894    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7e6d95a-83a3-44d5-9997-3fbbb01635a2" containerID="e7e8d3c4eea4116f40f6b728eb8b59616914f25e479feb9d742d8a349627b823" exitCode=0
Sep 10 12:57:43.774307 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:43.773988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z" event={"ID":"a7e6d95a-83a3-44d5-9997-3fbbb01635a2","Type":"ContainerDied","Data":"e7e8d3c4eea4116f40f6b728eb8b59616914f25e479feb9d742d8a349627b823"}
Sep 10 12:57:44.926125 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:44.926102    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"
Sep 10 12:57:45.072852 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:45.072751    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e6d95a-83a3-44d5-9997-3fbbb01635a2-maas-api-service-ca\") pod \"a7e6d95a-83a3-44d5-9997-3fbbb01635a2\" (UID: \"a7e6d95a-83a3-44d5-9997-3fbbb01635a2\") "
Sep 10 12:57:45.073084 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:45.073062    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e6d95a-83a3-44d5-9997-3fbbb01635a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7e6d95a-83a3-44d5-9997-3fbbb01635a2" (UID: "a7e6d95a-83a3-44d5-9997-3fbbb01635a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:57:45.174276 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:45.174240    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e6d95a-83a3-44d5-9997-3fbbb01635a2-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:57:45.786441 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:45.786410    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"
Sep 10 12:57:45.786683 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:45.786448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z" event={"ID":"a7e6d95a-83a3-44d5-9997-3fbbb01635a2","Type":"ContainerDied","Data":"b56fb90f2e829afdb5affe3afacd4c25b0b521e5bb953ce0b09d81ebced71b7e"}
Sep 10 12:57:45.786683 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:45.786514    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b56fb90f2e829afdb5affe3afacd4c25b0b521e5bb953ce0b09d81ebced71b7e"
Sep 10 12:57:47.087866 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:47.087832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:57.600385 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.600350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:57.654624 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.654575    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"]
Sep 10 12:57:57.654910 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.654853    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4" podUID="bbe8f292-3054-4bbd-84ba-e05a70a250b0" containerName="payload-pre-processing" containerID="cri-o://091713c3cbe380f1bba231b7c2b2fd884f924d29408ca474348c27127f9148a3" gracePeriod=30
Sep 10 12:57:57.711519 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.711487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:57.832321 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.832287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:57.858575 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.858500    2577 generic.go:356] "Generic (PLEG): container finished" podID="bbe8f292-3054-4bbd-84ba-e05a70a250b0" containerID="091713c3cbe380f1bba231b7c2b2fd884f924d29408ca474348c27127f9148a3" exitCode=0
Sep 10 12:57:57.858736 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.858555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4" event={"ID":"bbe8f292-3054-4bbd-84ba-e05a70a250b0","Type":"ContainerDied","Data":"091713c3cbe380f1bba231b7c2b2fd884f924d29408ca474348c27127f9148a3"}
Sep 10 12:57:57.905608 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.905577    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:57:57.943311 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.943279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:57.954294 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.954266    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"]
Sep 10 12:57:57.963708 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:57.963681    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe8579-r4s7z"]
Sep 10 12:57:58.001509 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.001476    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e6d95a-83a3-44d5-9997-3fbbb01635a2" path="/var/lib/kubelet/pods/a7e6d95a-83a3-44d5-9997-3fbbb01635a2/volumes"
Sep 10 12:57:58.006830 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.006810    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe8f292-3054-4bbd-84ba-e05a70a250b0-plugins-config-volume\") pod \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\" (UID: \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\") "
Sep 10 12:57:58.006933 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.006871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czstj\" (UniqueName: \"kubernetes.io/projected/bbe8f292-3054-4bbd-84ba-e05a70a250b0-kube-api-access-czstj\") pod \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\" (UID: \"bbe8f292-3054-4bbd-84ba-e05a70a250b0\") "
Sep 10 12:57:58.007136 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.007115    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe8f292-3054-4bbd-84ba-e05a70a250b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbe8f292-3054-4bbd-84ba-e05a70a250b0" (UID: "bbe8f292-3054-4bbd-84ba-e05a70a250b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:57:58.009335 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.009312    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe8f292-3054-4bbd-84ba-e05a70a250b0-kube-api-access-czstj" (OuterVolumeSpecName: "kube-api-access-czstj") pod "bbe8f292-3054-4bbd-84ba-e05a70a250b0" (UID: "bbe8f292-3054-4bbd-84ba-e05a70a250b0"). InnerVolumeSpecName "kube-api-access-czstj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:57:58.058679 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.058653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:58.108365 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.108338    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czstj\" (UniqueName: \"kubernetes.io/projected/bbe8f292-3054-4bbd-84ba-e05a70a250b0-kube-api-access-czstj\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:57:58.108365 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.108362    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe8f292-3054-4bbd-84ba-e05a70a250b0-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:57:58.410666 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.410624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:57:58.865265 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.865233    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"
Sep 10 12:57:58.865265 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.865261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4" event={"ID":"bbe8f292-3054-4bbd-84ba-e05a70a250b0","Type":"ContainerDied","Data":"7cd1a5695e27c31db74f7bf5d19e948a3ac81bef21e9f70b2b1a70062e0ee745"}
Sep 10 12:57:58.865772 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.865320    2577 scope.go:117] "RemoveContainer" containerID="091713c3cbe380f1bba231b7c2b2fd884f924d29408ca474348c27127f9148a3"
Sep 10 12:57:58.890543 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.890514    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"]
Sep 10 12:57:58.905771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:57:58.905744    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe8579-d599d4c54-rldb4"]
Sep 10 12:58:00.001482 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:00.001444    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe8f292-3054-4bbd-84ba-e05a70a250b0" path="/var/lib/kubelet/pods/bbe8f292-3054-4bbd-84ba-e05a70a250b0/volumes"
Sep 10 12:58:05.708676 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:05.708647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:05.865983 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:05.865948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:06.129678 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.129642    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:06.423776 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.423698    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"]
Sep 10 12:58:06.424321 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.424300    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e6d95a-83a3-44d5-9997-3fbbb01635a2" containerName="revoke-keys"
Sep 10 12:58:06.424321 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.424320    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e6d95a-83a3-44d5-9997-3fbbb01635a2" containerName="revoke-keys"
Sep 10 12:58:06.424436 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.424344    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe8f292-3054-4bbd-84ba-e05a70a250b0" containerName="payload-pre-processing"
Sep 10 12:58:06.424436 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.424350    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe8f292-3054-4bbd-84ba-e05a70a250b0" containerName="payload-pre-processing"
Sep 10 12:58:06.424502 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.424439    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e6d95a-83a3-44d5-9997-3fbbb01635a2" containerName="revoke-keys"
Sep 10 12:58:06.424502 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.424457    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe8f292-3054-4bbd-84ba-e05a70a250b0" containerName="payload-pre-processing"
Sep 10 12:58:06.428842 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.428823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"
Sep 10 12:58:06.432454 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.432431    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:58:06.442546 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.442513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"]
Sep 10 12:58:06.491923 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.491887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72621be-2166-4074-8e08-def2f59ac37b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw\" (UID: \"b72621be-2166-4074-8e08-def2f59ac37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"
Sep 10 12:58:06.592465 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.592434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72621be-2166-4074-8e08-def2f59ac37b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw\" (UID: \"b72621be-2166-4074-8e08-def2f59ac37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"
Sep 10 12:58:06.593134 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.593113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72621be-2166-4074-8e08-def2f59ac37b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw\" (UID: \"b72621be-2166-4074-8e08-def2f59ac37b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"
Sep 10 12:58:06.599394 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.599371    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:06.739264 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:06.739187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"
Sep 10 12:58:07.015364 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:07.015336    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"]
Sep 10 12:58:07.020056 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:58:07.020025    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72621be_2166_4074_8e08_def2f59ac37b.slice/crio-c1987be50f9c895b40a43cb30a827ac4c6461fe1f5b4c4a4d672dd0d58cda548 WatchSource:0}: Error finding container c1987be50f9c895b40a43cb30a827ac4c6461fe1f5b4c4a4d672dd0d58cda548: Status 404 returned error can't find the container with id c1987be50f9c895b40a43cb30a827ac4c6461fe1f5b4c4a4d672dd0d58cda548
Sep 10 12:58:07.928804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:07.928772    2577 generic.go:356] "Generic (PLEG): container finished" podID="b72621be-2166-4074-8e08-def2f59ac37b" containerID="b213c7039796ff95c49554b102a16ef2f827ffd2dd9aa764bb4e515d1b939aaf" exitCode=0
Sep 10 12:58:07.928804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:07.928808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw" event={"ID":"b72621be-2166-4074-8e08-def2f59ac37b","Type":"ContainerDied","Data":"b213c7039796ff95c49554b102a16ef2f827ffd2dd9aa764bb4e515d1b939aaf"}
Sep 10 12:58:07.929216 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:07.928844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw" event={"ID":"b72621be-2166-4074-8e08-def2f59ac37b","Type":"ContainerStarted","Data":"c1987be50f9c895b40a43cb30a827ac4c6461fe1f5b4c4a4d672dd0d58cda548"}
Sep 10 12:58:09.080544 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:09.080519    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"
Sep 10 12:58:09.120154 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:09.120125    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72621be-2166-4074-8e08-def2f59ac37b-maas-api-service-ca\") pod \"b72621be-2166-4074-8e08-def2f59ac37b\" (UID: \"b72621be-2166-4074-8e08-def2f59ac37b\") "
Sep 10 12:58:09.120481 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:09.120457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72621be-2166-4074-8e08-def2f59ac37b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b72621be-2166-4074-8e08-def2f59ac37b" (UID: "b72621be-2166-4074-8e08-def2f59ac37b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:09.221270 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:09.221185    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72621be-2166-4074-8e08-def2f59ac37b-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:09.941467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:09.941425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw" event={"ID":"b72621be-2166-4074-8e08-def2f59ac37b","Type":"ContainerDied","Data":"c1987be50f9c895b40a43cb30a827ac4c6461fe1f5b4c4a4d672dd0d58cda548"}
Sep 10 12:58:09.941467 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:09.941445    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"
Sep 10 12:58:09.941780 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:09.941477    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1987be50f9c895b40a43cb30a827ac4c6461fe1f5b4c4a4d672dd0d58cda548"
Sep 10 12:58:22.032585 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.032550    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:22.083192 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.083155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"]
Sep 10 12:58:22.083451 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.083414    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9" podUID="a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1" containerName="payload-processing" containerID="cri-o://a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9" gracePeriod=30
Sep 10 12:58:22.092049 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.092014    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"]
Sep 10 12:58:22.092239 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.092214    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd" podUID="e5747814-5c17-4bc2-a167-66d95a8a9de1" containerName="payload-pre-processing" containerID="cri-o://a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8" gracePeriod=30
Sep 10 12:58:22.133075 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.133046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:22.235049 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.235014    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:22.334742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.334706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:22.350170 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.350147    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:58:22.370436 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.370410    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"]
Sep 10 12:58:22.372733 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.372713    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3c07b-6cprw"]
Sep 10 12:58:22.383641 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.383581    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:58:22.425111 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.425081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:22.453850 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.453820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56kbc\" (UniqueName: \"kubernetes.io/projected/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-kube-api-access-56kbc\") pod \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\" (UID: \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\") "
Sep 10 12:58:22.454011 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.453862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-plugins-config-volume\") pod \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\" (UID: \"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1\") "
Sep 10 12:58:22.454231 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.454210    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1" (UID: "a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:22.455965 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.455945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-kube-api-access-56kbc" (OuterVolumeSpecName: "kube-api-access-56kbc") pod "a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1" (UID: "a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1"). InnerVolumeSpecName "kube-api-access-56kbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:22.555137 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.555099    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5747814-5c17-4bc2-a167-66d95a8a9de1-plugins-config-volume\") pod \"e5747814-5c17-4bc2-a167-66d95a8a9de1\" (UID: \"e5747814-5c17-4bc2-a167-66d95a8a9de1\") "
Sep 10 12:58:22.555361 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.555200    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jxzz\" (UniqueName: \"kubernetes.io/projected/e5747814-5c17-4bc2-a167-66d95a8a9de1-kube-api-access-9jxzz\") pod \"e5747814-5c17-4bc2-a167-66d95a8a9de1\" (UID: \"e5747814-5c17-4bc2-a167-66d95a8a9de1\") "
Sep 10 12:58:22.555516 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.555494    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56kbc\" (UniqueName: \"kubernetes.io/projected/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-kube-api-access-56kbc\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:22.555587 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.555522    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:22.555587 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.555537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5747814-5c17-4bc2-a167-66d95a8a9de1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5747814-5c17-4bc2-a167-66d95a8a9de1" (UID: "e5747814-5c17-4bc2-a167-66d95a8a9de1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:22.557279 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.557258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5747814-5c17-4bc2-a167-66d95a8a9de1-kube-api-access-9jxzz" (OuterVolumeSpecName: "kube-api-access-9jxzz") pod "e5747814-5c17-4bc2-a167-66d95a8a9de1" (UID: "e5747814-5c17-4bc2-a167-66d95a8a9de1"). InnerVolumeSpecName "kube-api-access-9jxzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:22.656759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.656730    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5747814-5c17-4bc2-a167-66d95a8a9de1-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:22.656759 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.656753    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jxzz\" (UniqueName: \"kubernetes.io/projected/e5747814-5c17-4bc2-a167-66d95a8a9de1-kube-api-access-9jxzz\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:22.816561 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:22.816460    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:23.017672 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.017632    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5747814-5c17-4bc2-a167-66d95a8a9de1" containerID="a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8" exitCode=0
Sep 10 12:58:23.017853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.017676    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"
Sep 10 12:58:23.017853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.017718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd" event={"ID":"e5747814-5c17-4bc2-a167-66d95a8a9de1","Type":"ContainerDied","Data":"a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8"}
Sep 10 12:58:23.017853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.017768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd" event={"ID":"e5747814-5c17-4bc2-a167-66d95a8a9de1","Type":"ContainerDied","Data":"0dbaafb9f3ea265ebccb783b2870e69eb747ce6f713f220e923d30202435c100"}
Sep 10 12:58:23.017853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.017801    2577 scope.go:117] "RemoveContainer" containerID="a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8"
Sep 10 12:58:23.019308 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.019282    2577 generic.go:356] "Generic (PLEG): container finished" podID="a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1" containerID="a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9" exitCode=0
Sep 10 12:58:23.019427 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.019315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9" event={"ID":"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1","Type":"ContainerDied","Data":"a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9"}
Sep 10 12:58:23.019427 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.019367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9" event={"ID":"a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1","Type":"ContainerDied","Data":"f0c55ce7f6d9da086816111157f9bcc3cff847d4609ad20bffb2bf8b2aa8d46b"}
Sep 10 12:58:23.019427 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.019329    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"
Sep 10 12:58:23.029999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.029163    2577 scope.go:117] "RemoveContainer" containerID="a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8"
Sep 10 12:58:23.029999 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:58:23.029875    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8\": container with ID starting with a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8 not found: ID does not exist" containerID="a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8"
Sep 10 12:58:23.029999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.029908    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8"} err="failed to get container status \"a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8\": rpc error: code = NotFound desc = could not find container \"a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8\": container with ID starting with a7cbf799b49cd10a265633062314653c41994ec438a4a3f06dd8323f9235e6d8 not found: ID does not exist"
Sep 10 12:58:23.029999 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.029930    2577 scope.go:117] "RemoveContainer" containerID="a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9"
Sep 10 12:58:23.038862 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.038845    2577 scope.go:117] "RemoveContainer" containerID="a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9"
Sep 10 12:58:23.039099 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:58:23.039085    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9\": container with ID starting with a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9 not found: ID does not exist" containerID="a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9"
Sep 10 12:58:23.039143 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.039109    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9"} err="failed to get container status \"a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9\": rpc error: code = NotFound desc = could not find container \"a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9\": container with ID starting with a19e953d9614b474b2082a179022bf1c25a5b6768728072b62aa3af57be83ec9 not found: ID does not exist"
Sep 10 12:58:23.054807 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.054786    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"]
Sep 10 12:58:23.067911 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.061117    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3c07b-7dccc46b59-6rht9"]
Sep 10 12:58:23.080986 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.080960    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"]
Sep 10 12:58:23.083036 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:23.083015    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3c07b-8485b7b5f4-4xjqd"]
Sep 10 12:58:24.001478 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:24.001440    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1" path="/var/lib/kubelet/pods/a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1/volumes"
Sep 10 12:58:24.001845 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:24.001831    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72621be-2166-4074-8e08-def2f59ac37b" path="/var/lib/kubelet/pods/b72621be-2166-4074-8e08-def2f59ac37b/volumes"
Sep 10 12:58:24.002158 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:24.002145    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5747814-5c17-4bc2-a167-66d95a8a9de1" path="/var/lib/kubelet/pods/e5747814-5c17-4bc2-a167-66d95a8a9de1/volumes"
Sep 10 12:58:24.461920 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:24.461887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:29.507605 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:29.507572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:29.910308 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:29.910272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:30.004200 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.004172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:30.159146 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.159110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:30.189112 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189032    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"]
Sep 10 12:58:30.189543 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189531    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72621be-2166-4074-8e08-def2f59ac37b" containerName="revoke-keys"
Sep 10 12:58:30.189614 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189545    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72621be-2166-4074-8e08-def2f59ac37b" containerName="revoke-keys"
Sep 10 12:58:30.189614 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189554    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1" containerName="payload-processing"
Sep 10 12:58:30.189614 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189560    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1" containerName="payload-processing"
Sep 10 12:58:30.189614 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189568    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5747814-5c17-4bc2-a167-66d95a8a9de1" containerName="payload-pre-processing"
Sep 10 12:58:30.189614 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189573    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5747814-5c17-4bc2-a167-66d95a8a9de1" containerName="payload-pre-processing"
Sep 10 12:58:30.189804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189671    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f15272-9bb4-41f2-9e9c-d674ab5a2fc1" containerName="payload-processing"
Sep 10 12:58:30.189804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189679    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72621be-2166-4074-8e08-def2f59ac37b" containerName="revoke-keys"
Sep 10 12:58:30.189804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.189688    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5747814-5c17-4bc2-a167-66d95a8a9de1" containerName="payload-pre-processing"
Sep 10 12:58:30.192946 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.192926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"
Sep 10 12:58:30.196625 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.196586    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:58:30.206524 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.206504    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"]
Sep 10 12:58:30.337340 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.337310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a96695-02cc-4e64-8db3-36e6e3ae1d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms\" (UID: \"42a96695-02cc-4e64-8db3-36e6e3ae1d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"
Sep 10 12:58:30.438934 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.438893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a96695-02cc-4e64-8db3-36e6e3ae1d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms\" (UID: \"42a96695-02cc-4e64-8db3-36e6e3ae1d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"
Sep 10 12:58:30.439537 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.439492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a96695-02cc-4e64-8db3-36e6e3ae1d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms\" (UID: \"42a96695-02cc-4e64-8db3-36e6e3ae1d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"
Sep 10 12:58:30.503036 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.502998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"
Sep 10 12:58:30.508799 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.508774    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:30.801250 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:30.801213    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"]
Sep 10 12:58:30.837463 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:58:30.837436    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a96695_02cc_4e64_8db3_36e6e3ae1d82.slice/crio-adf5cec94600a0a96dae705543e1f904de2bf3109f40d055a096cb1683c57da9 WatchSource:0}: Error finding container adf5cec94600a0a96dae705543e1f904de2bf3109f40d055a096cb1683c57da9: Status 404 returned error can't find the container with id adf5cec94600a0a96dae705543e1f904de2bf3109f40d055a096cb1683c57da9
Sep 10 12:58:31.067993 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:31.067914    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms" event={"ID":"42a96695-02cc-4e64-8db3-36e6e3ae1d82","Type":"ContainerStarted","Data":"807699fddd1f9e9acb4b1e8664f561b84b09aa161f0f273ed620c832444487b9"}
Sep 10 12:58:31.067993 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:31.067966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms" event={"ID":"42a96695-02cc-4e64-8db3-36e6e3ae1d82","Type":"ContainerStarted","Data":"adf5cec94600a0a96dae705543e1f904de2bf3109f40d055a096cb1683c57da9"}
Sep 10 12:58:31.096004 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:31.095947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms" podStartSLOduration=1.095930761 podStartE2EDuration="1.095930761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:58:31.093491409 +0000 UTC m=+1685.615660461" watchObservedRunningTime="2026-09-10 12:58:31.095930761 +0000 UTC m=+1685.618099814"
Sep 10 12:58:31.906190 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:31.906092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:32.075828 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:32.075790    2577 generic.go:356] "Generic (PLEG): container finished" podID="42a96695-02cc-4e64-8db3-36e6e3ae1d82" containerID="807699fddd1f9e9acb4b1e8664f561b84b09aa161f0f273ed620c832444487b9" exitCode=0
Sep 10 12:58:32.075991 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:32.075870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms" event={"ID":"42a96695-02cc-4e64-8db3-36e6e3ae1d82","Type":"ContainerDied","Data":"807699fddd1f9e9acb4b1e8664f561b84b09aa161f0f273ed620c832444487b9"}
Sep 10 12:58:32.699981 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:32.699955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:33.218106 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:33.218081    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"
Sep 10 12:58:33.262136 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:33.262110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a96695-02cc-4e64-8db3-36e6e3ae1d82-maas-api-service-ca\") pod \"42a96695-02cc-4e64-8db3-36e6e3ae1d82\" (UID: \"42a96695-02cc-4e64-8db3-36e6e3ae1d82\") "
Sep 10 12:58:33.262491 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:33.262467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a96695-02cc-4e64-8db3-36e6e3ae1d82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42a96695-02cc-4e64-8db3-36e6e3ae1d82" (UID: "42a96695-02cc-4e64-8db3-36e6e3ae1d82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:33.363191 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:33.363114    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a96695-02cc-4e64-8db3-36e6e3ae1d82-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:34.088011 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:34.087978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms" event={"ID":"42a96695-02cc-4e64-8db3-36e6e3ae1d82","Type":"ContainerDied","Data":"adf5cec94600a0a96dae705543e1f904de2bf3109f40d055a096cb1683c57da9"}
Sep 10 12:58:34.088164 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:34.088031    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adf5cec94600a0a96dae705543e1f904de2bf3109f40d055a096cb1683c57da9"
Sep 10 12:58:34.088164 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:34.087991    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"
Sep 10 12:58:35.125654 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:35.125623    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"]
Sep 10 12:58:35.283541 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:35.283507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:41.411579 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.411525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:41.423177 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.423146    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"]
Sep 10 12:58:41.423424 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.423392    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck" podUID="acea910d-8e18-4ab3-b607-17130a969fc9" containerName="payload-processing" containerID="cri-o://e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223" gracePeriod=30
Sep 10 12:58:41.432419 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.432393    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"]
Sep 10 12:58:41.432622 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.432579    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r" podUID="259e6e80-ae96-48b6-a303-794601fdbdaa" containerName="payload-pre-processing" containerID="cri-o://0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8" gracePeriod=30
Sep 10 12:58:41.541477 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.541442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:41.719877 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.719849    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:58:41.722900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.722879    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:58:41.739830 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.739810    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6e80-ae96-48b6-a303-794601fdbdaa-plugins-config-volume\") pod \"259e6e80-ae96-48b6-a303-794601fdbdaa\" (UID: \"259e6e80-ae96-48b6-a303-794601fdbdaa\") "
Sep 10 12:58:41.739949 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.739856    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86fpp\" (UniqueName: \"kubernetes.io/projected/acea910d-8e18-4ab3-b607-17130a969fc9-kube-api-access-86fpp\") pod \"acea910d-8e18-4ab3-b607-17130a969fc9\" (UID: \"acea910d-8e18-4ab3-b607-17130a969fc9\") "
Sep 10 12:58:41.739949 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.739905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mnkg\" (UniqueName: \"kubernetes.io/projected/259e6e80-ae96-48b6-a303-794601fdbdaa-kube-api-access-8mnkg\") pod \"259e6e80-ae96-48b6-a303-794601fdbdaa\" (UID: \"259e6e80-ae96-48b6-a303-794601fdbdaa\") "
Sep 10 12:58:41.740068 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.740006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acea910d-8e18-4ab3-b607-17130a969fc9-plugins-config-volume\") pod \"acea910d-8e18-4ab3-b607-17130a969fc9\" (UID: \"acea910d-8e18-4ab3-b607-17130a969fc9\") "
Sep 10 12:58:41.740232 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.740207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259e6e80-ae96-48b6-a303-794601fdbdaa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "259e6e80-ae96-48b6-a303-794601fdbdaa" (UID: "259e6e80-ae96-48b6-a303-794601fdbdaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:41.740449 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.740426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acea910d-8e18-4ab3-b607-17130a969fc9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acea910d-8e18-4ab3-b607-17130a969fc9" (UID: "acea910d-8e18-4ab3-b607-17130a969fc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:41.742063 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.742042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acea910d-8e18-4ab3-b607-17130a969fc9-kube-api-access-86fpp" (OuterVolumeSpecName: "kube-api-access-86fpp") pod "acea910d-8e18-4ab3-b607-17130a969fc9" (UID: "acea910d-8e18-4ab3-b607-17130a969fc9"). InnerVolumeSpecName "kube-api-access-86fpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:41.742133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.742066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259e6e80-ae96-48b6-a303-794601fdbdaa-kube-api-access-8mnkg" (OuterVolumeSpecName: "kube-api-access-8mnkg") pod "259e6e80-ae96-48b6-a303-794601fdbdaa" (UID: "259e6e80-ae96-48b6-a303-794601fdbdaa"). InnerVolumeSpecName "kube-api-access-8mnkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:41.823133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.823086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:41.841524 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.841489    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acea910d-8e18-4ab3-b607-17130a969fc9-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:41.841524 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.841518    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e6e80-ae96-48b6-a303-794601fdbdaa-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:41.841524 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.841529    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86fpp\" (UniqueName: \"kubernetes.io/projected/acea910d-8e18-4ab3-b607-17130a969fc9-kube-api-access-86fpp\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:41.841771 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.841538    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mnkg\" (UniqueName: \"kubernetes.io/projected/259e6e80-ae96-48b6-a303-794601fdbdaa-kube-api-access-8mnkg\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:41.913970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.913928    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"]
Sep 10 12:58:41.925533 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.925502    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ac8db6-64zms"]
Sep 10 12:58:41.976541 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:41.976456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:42.000969 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.000932    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a96695-02cc-4e64-8db3-36e6e3ae1d82" path="/var/lib/kubelet/pods/42a96695-02cc-4e64-8db3-36e6e3ae1d82/volumes"
Sep 10 12:58:42.132538 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.132506    2577 generic.go:356] "Generic (PLEG): container finished" podID="acea910d-8e18-4ab3-b607-17130a969fc9" containerID="e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223" exitCode=0
Sep 10 12:58:42.132719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.132553    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"
Sep 10 12:58:42.132719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.132585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck" event={"ID":"acea910d-8e18-4ab3-b607-17130a969fc9","Type":"ContainerDied","Data":"e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223"}
Sep 10 12:58:42.132719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.132656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck" event={"ID":"acea910d-8e18-4ab3-b607-17130a969fc9","Type":"ContainerDied","Data":"2f1d54e9eb1cff198b5a171c8a9ebc41afc03c4938563fcfb6b071811e5ee5ce"}
Sep 10 12:58:42.132719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.132681    2577 scope.go:117] "RemoveContainer" containerID="e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223"
Sep 10 12:58:42.134195 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.134166    2577 generic.go:356] "Generic (PLEG): container finished" podID="259e6e80-ae96-48b6-a303-794601fdbdaa" containerID="0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8" exitCode=0
Sep 10 12:58:42.134296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.134219    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"
Sep 10 12:58:42.134296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.134223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r" event={"ID":"259e6e80-ae96-48b6-a303-794601fdbdaa","Type":"ContainerDied","Data":"0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8"}
Sep 10 12:58:42.134296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.134266    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r" event={"ID":"259e6e80-ae96-48b6-a303-794601fdbdaa","Type":"ContainerDied","Data":"4b6906753c401225b82e6012324976592ea315cfee40e0f5e2365daedfaf122c"}
Sep 10 12:58:42.142171 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.142153    2577 scope.go:117] "RemoveContainer" containerID="e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223"
Sep 10 12:58:42.142442 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:58:42.142424    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223\": container with ID starting with e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223 not found: ID does not exist" containerID="e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223"
Sep 10 12:58:42.142525 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.142454    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223"} err="failed to get container status \"e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223\": rpc error: code = NotFound desc = could not find container \"e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223\": container with ID starting with e3ad8c954d576aacd827d74778fb07ff09c8d3dd192da6f894e40c3f8bcde223 not found: ID does not exist"
Sep 10 12:58:42.142525 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.142479    2577 scope.go:117] "RemoveContainer" containerID="0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8"
Sep 10 12:58:42.150764 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.150746    2577 scope.go:117] "RemoveContainer" containerID="0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8"
Sep 10 12:58:42.151048 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:58:42.151031    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8\": container with ID starting with 0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8 not found: ID does not exist" containerID="0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8"
Sep 10 12:58:42.151121 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.151058    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8"} err="failed to get container status \"0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8\": rpc error: code = NotFound desc = could not find container \"0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8\": container with ID starting with 0781decf6eaad6d84435f22c432acb28d8a343319f4d72bdd4096fb722cfe4f8 not found: ID does not exist"
Sep 10 12:58:42.162082 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.162056    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"]
Sep 10 12:58:42.168848 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.168828    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ac8db6-5fb745fff8-tp78r"]
Sep 10 12:58:42.197944 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.197918    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"]
Sep 10 12:58:42.209240 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.209221    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac8db6-5789f59d99-rkxck"]
Sep 10 12:58:42.314259 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:42.314223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:44.002504 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:44.002467    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="259e6e80-ae96-48b6-a303-794601fdbdaa" path="/var/lib/kubelet/pods/259e6e80-ae96-48b6-a303-794601fdbdaa/volumes"
Sep 10 12:58:44.003069 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:44.003048    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acea910d-8e18-4ab3-b607-17130a969fc9" path="/var/lib/kubelet/pods/acea910d-8e18-4ab3-b607-17130a969fc9/volumes"
Sep 10 12:58:46.353150 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.353104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:46.374234 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.374197    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"]
Sep 10 12:58:46.376404 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.376380    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1312ea-29817405-spnrb"]
Sep 10 12:58:46.414891 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.414858    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"]
Sep 10 12:58:46.415128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.415095    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l" podUID="1d31dcec-e6d9-4c53-a925-c05a14b67754" containerName="payload-processing" containerID="cri-o://c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd" gracePeriod=30
Sep 10 12:58:46.422774 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.422746    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"]
Sep 10 12:58:46.423736 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.422990    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4" podUID="d8727ee6-1393-4331-b6b0-b7f5f3a25f39" containerName="payload-pre-processing" containerID="cri-o://ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3" gracePeriod=30
Sep 10 12:58:46.500023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.499992    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:46.603783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.603697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:46.707297 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.707274    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:58:46.710287 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.710270    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:58:46.785873 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.785844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm88k\" (UniqueName: \"kubernetes.io/projected/1d31dcec-e6d9-4c53-a925-c05a14b67754-kube-api-access-tm88k\") pod \"1d31dcec-e6d9-4c53-a925-c05a14b67754\" (UID: \"1d31dcec-e6d9-4c53-a925-c05a14b67754\") "
Sep 10 12:58:46.786023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.785891    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d31dcec-e6d9-4c53-a925-c05a14b67754-plugins-config-volume\") pod \"1d31dcec-e6d9-4c53-a925-c05a14b67754\" (UID: \"1d31dcec-e6d9-4c53-a925-c05a14b67754\") "
Sep 10 12:58:46.786023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.785934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-plugins-config-volume\") pod \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\" (UID: \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\") "
Sep 10 12:58:46.786023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.786005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr48x\" (UniqueName: \"kubernetes.io/projected/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-kube-api-access-cr48x\") pod \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\" (UID: \"d8727ee6-1393-4331-b6b0-b7f5f3a25f39\") "
Sep 10 12:58:46.786319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.786296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d31dcec-e6d9-4c53-a925-c05a14b67754-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d31dcec-e6d9-4c53-a925-c05a14b67754" (UID: "1d31dcec-e6d9-4c53-a925-c05a14b67754"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:46.786404 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.786361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8727ee6-1393-4331-b6b0-b7f5f3a25f39" (UID: "d8727ee6-1393-4331-b6b0-b7f5f3a25f39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:46.788001 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.787983    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d31dcec-e6d9-4c53-a925-c05a14b67754-kube-api-access-tm88k" (OuterVolumeSpecName: "kube-api-access-tm88k") pod "1d31dcec-e6d9-4c53-a925-c05a14b67754" (UID: "1d31dcec-e6d9-4c53-a925-c05a14b67754"). InnerVolumeSpecName "kube-api-access-tm88k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:46.788115 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.788094    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-kube-api-access-cr48x" (OuterVolumeSpecName: "kube-api-access-cr48x") pod "d8727ee6-1393-4331-b6b0-b7f5f3a25f39" (UID: "d8727ee6-1393-4331-b6b0-b7f5f3a25f39"). InnerVolumeSpecName "kube-api-access-cr48x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:46.797075 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.797048    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:46.887556 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.887493    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm88k\" (UniqueName: \"kubernetes.io/projected/1d31dcec-e6d9-4c53-a925-c05a14b67754-kube-api-access-tm88k\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:46.887556 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.887517    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d31dcec-e6d9-4c53-a925-c05a14b67754-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:46.887556 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.887527    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:46.887556 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:46.887539    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr48x\" (UniqueName: \"kubernetes.io/projected/d8727ee6-1393-4331-b6b0-b7f5f3a25f39-kube-api-access-cr48x\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:47.066823 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.066782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:47.166295 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.166209    2577 generic.go:356] "Generic (PLEG): container finished" podID="d8727ee6-1393-4331-b6b0-b7f5f3a25f39" containerID="ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3" exitCode=0
Sep 10 12:58:47.166295 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.166259    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"
Sep 10 12:58:47.166496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.166253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4" event={"ID":"d8727ee6-1393-4331-b6b0-b7f5f3a25f39","Type":"ContainerDied","Data":"ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3"}
Sep 10 12:58:47.166496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.166375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4" event={"ID":"d8727ee6-1393-4331-b6b0-b7f5f3a25f39","Type":"ContainerDied","Data":"5037d1d91e55e54520ca2fc1d6c2c3772a96be56c46c3b4554355440ef4f96f7"}
Sep 10 12:58:47.166496 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.166397    2577 scope.go:117] "RemoveContainer" containerID="ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3"
Sep 10 12:58:47.168013 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.167988    2577 generic.go:356] "Generic (PLEG): container finished" podID="1d31dcec-e6d9-4c53-a925-c05a14b67754" containerID="c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd" exitCode=0
Sep 10 12:58:47.168124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.168011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l" event={"ID":"1d31dcec-e6d9-4c53-a925-c05a14b67754","Type":"ContainerDied","Data":"c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd"}
Sep 10 12:58:47.168124 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.168045    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"
Sep 10 12:58:47.168261 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.168052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l" event={"ID":"1d31dcec-e6d9-4c53-a925-c05a14b67754","Type":"ContainerDied","Data":"323d11b8617735876e199dfe8dd7aa510771e1bd37140bacf83e920f7074e2cd"}
Sep 10 12:58:47.177336 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.177308    2577 scope.go:117] "RemoveContainer" containerID="ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3"
Sep 10 12:58:47.177654 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:58:47.177632    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3\": container with ID starting with ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3 not found: ID does not exist" containerID="ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3"
Sep 10 12:58:47.177740 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.177663    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3"} err="failed to get container status \"ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3\": rpc error: code = NotFound desc = could not find container \"ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3\": container with ID starting with ef07b443aaa814eb75785fac4960fcdea7e3f0a896c77e5416527c771d6618c3 not found: ID does not exist"
Sep 10 12:58:47.177740 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.177684    2577 scope.go:117] "RemoveContainer" containerID="c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd"
Sep 10 12:58:47.187366 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.187350    2577 scope.go:117] "RemoveContainer" containerID="c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd"
Sep 10 12:58:47.187664 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:58:47.187639    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd\": container with ID starting with c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd not found: ID does not exist" containerID="c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd"
Sep 10 12:58:47.187719 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.187675    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd"} err="failed to get container status \"c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd\": rpc error: code = NotFound desc = could not find container \"c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd\": container with ID starting with c0a259a8f0d17a74bd86121a1ce040dcf8b1b8978647897448f1636c31c2d1dd not found: ID does not exist"
Sep 10 12:58:47.220896 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.220872    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"]
Sep 10 12:58:47.232801 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.232781    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1312ea-6579d7db85-x7j5l"]
Sep 10 12:58:47.296933 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.296905    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"]
Sep 10 12:58:47.321878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:47.321852    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1312ea-5fc98968cd-bthc4"]
Sep 10 12:58:48.001838 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:48.001755    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d31dcec-e6d9-4c53-a925-c05a14b67754" path="/var/lib/kubelet/pods/1d31dcec-e6d9-4c53-a925-c05a14b67754/volumes"
Sep 10 12:58:48.002236 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:48.002108    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8727ee6-1393-4331-b6b0-b7f5f3a25f39" path="/var/lib/kubelet/pods/d8727ee6-1393-4331-b6b0-b7f5f3a25f39/volumes"
Sep 10 12:58:48.002346 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:48.002332    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e044bb97-066e-4aae-886c-b9790dfd6bff" path="/var/lib/kubelet/pods/e044bb97-066e-4aae-886c-b9790dfd6bff/volumes"
Sep 10 12:58:50.118925 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.118877    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:50.145950 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.145909    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48" podUID="763ee82a-1a3c-4bd9-8315-61e7fc00d191" containerName="main" containerID="cri-o://42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014" gracePeriod=45
Sep 10 12:58:50.219077 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.219024    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:50.410540 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.410383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:50.445890 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.445861    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:58:50.521237 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-model-cache\") pod \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") "
Sep 10 12:58:50.521416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521262    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-home\") pod \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") "
Sep 10 12:58:50.521416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tmp-dir\") pod \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") "
Sep 10 12:58:50.521416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-dshm\") pod \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") "
Sep 10 12:58:50.521416 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tls-certs\") pod \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") "
Sep 10 12:58:50.521655 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521440    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr42k\" (UniqueName: \"kubernetes.io/projected/763ee82a-1a3c-4bd9-8315-61e7fc00d191-kube-api-access-kr42k\") pod \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\" (UID: \"763ee82a-1a3c-4bd9-8315-61e7fc00d191\") "
Sep 10 12:58:50.521655 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-model-cache" (OuterVolumeSpecName: "model-cache") pod "763ee82a-1a3c-4bd9-8315-61e7fc00d191" (UID: "763ee82a-1a3c-4bd9-8315-61e7fc00d191"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.521655 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-home" (OuterVolumeSpecName: "home") pod "763ee82a-1a3c-4bd9-8315-61e7fc00d191" (UID: "763ee82a-1a3c-4bd9-8315-61e7fc00d191"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.521655 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521574    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "763ee82a-1a3c-4bd9-8315-61e7fc00d191" (UID: "763ee82a-1a3c-4bd9-8315-61e7fc00d191"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.521878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521759    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.521878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521779    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.521878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.521793    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.523736 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.523711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763ee82a-1a3c-4bd9-8315-61e7fc00d191-kube-api-access-kr42k" (OuterVolumeSpecName: "kube-api-access-kr42k") pod "763ee82a-1a3c-4bd9-8315-61e7fc00d191" (UID: "763ee82a-1a3c-4bd9-8315-61e7fc00d191"). InnerVolumeSpecName "kube-api-access-kr42k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:50.523853 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.523831    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "763ee82a-1a3c-4bd9-8315-61e7fc00d191" (UID: "763ee82a-1a3c-4bd9-8315-61e7fc00d191"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:58:50.523921 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.523902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-dshm" (OuterVolumeSpecName: "dshm") pod "763ee82a-1a3c-4bd9-8315-61e7fc00d191" (UID: "763ee82a-1a3c-4bd9-8315-61e7fc00d191"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.622775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.622739    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/763ee82a-1a3c-4bd9-8315-61e7fc00d191-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.622775 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.622775    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr42k\" (UniqueName: \"kubernetes.io/projected/763ee82a-1a3c-4bd9-8315-61e7fc00d191-kube-api-access-kr42k\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.622953 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.622787    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/763ee82a-1a3c-4bd9-8315-61e7fc00d191-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.747568 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.747486    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"]
Sep 10 12:58:50.748037 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748018    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259e6e80-ae96-48b6-a303-794601fdbdaa" containerName="payload-pre-processing"
Sep 10 12:58:50.748037 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748036    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="259e6e80-ae96-48b6-a303-794601fdbdaa" containerName="payload-pre-processing"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748046    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a96695-02cc-4e64-8db3-36e6e3ae1d82" containerName="revoke-keys"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748052    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a96695-02cc-4e64-8db3-36e6e3ae1d82" containerName="revoke-keys"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748068    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acea910d-8e18-4ab3-b607-17130a969fc9" containerName="payload-processing"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748074    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="acea910d-8e18-4ab3-b607-17130a969fc9" containerName="payload-processing"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748089    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8727ee6-1393-4331-b6b0-b7f5f3a25f39" containerName="payload-pre-processing"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748094    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8727ee6-1393-4331-b6b0-b7f5f3a25f39" containerName="payload-pre-processing"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748100    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="763ee82a-1a3c-4bd9-8315-61e7fc00d191" containerName="main"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748105    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="763ee82a-1a3c-4bd9-8315-61e7fc00d191" containerName="main"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748119    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d31dcec-e6d9-4c53-a925-c05a14b67754" containerName="payload-processing"
Sep 10 12:58:50.748178 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748127    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d31dcec-e6d9-4c53-a925-c05a14b67754" containerName="payload-processing"
Sep 10 12:58:50.748479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748197    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8727ee6-1393-4331-b6b0-b7f5f3a25f39" containerName="payload-pre-processing"
Sep 10 12:58:50.748479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748210    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d31dcec-e6d9-4c53-a925-c05a14b67754" containerName="payload-processing"
Sep 10 12:58:50.748479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748220    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a96695-02cc-4e64-8db3-36e6e3ae1d82" containerName="revoke-keys"
Sep 10 12:58:50.748479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748232    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="259e6e80-ae96-48b6-a303-794601fdbdaa" containerName="payload-pre-processing"
Sep 10 12:58:50.748479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748243    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="763ee82a-1a3c-4bd9-8315-61e7fc00d191" containerName="main"
Sep 10 12:58:50.748479 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.748249    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="acea910d-8e18-4ab3-b607-17130a969fc9" containerName="payload-processing"
Sep 10 12:58:50.751322 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.751307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"
Sep 10 12:58:50.754335 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.754314    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:58:50.762323 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.762296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"]
Sep 10 12:58:50.824697 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.824673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd118765-ffb5-423e-9202-d59d02339b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz\" (UID: \"cd118765-ffb5-423e-9202-d59d02339b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"
Sep 10 12:58:50.925373 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.925333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd118765-ffb5-423e-9202-d59d02339b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz\" (UID: \"cd118765-ffb5-423e-9202-d59d02339b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"
Sep 10 12:58:50.925953 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:50.925932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd118765-ffb5-423e-9202-d59d02339b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz\" (UID: \"cd118765-ffb5-423e-9202-d59d02339b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"
Sep 10 12:58:51.007162 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.007080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:51.061822 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.061797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"
Sep 10 12:58:51.193526 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.193492    2577 generic.go:356] "Generic (PLEG): container finished" podID="763ee82a-1a3c-4bd9-8315-61e7fc00d191" containerID="42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014" exitCode=0
Sep 10 12:58:51.193964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.193538    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"
Sep 10 12:58:51.193964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.193577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48" event={"ID":"763ee82a-1a3c-4bd9-8315-61e7fc00d191","Type":"ContainerDied","Data":"42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014"}
Sep 10 12:58:51.193964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.193640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48" event={"ID":"763ee82a-1a3c-4bd9-8315-61e7fc00d191","Type":"ContainerDied","Data":"6d8167e9eba866910eedcd63382e9b0bad52054bf0266c0843f8cbbde21dc174"}
Sep 10 12:58:51.193964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.193660    2577 scope.go:117] "RemoveContainer" containerID="42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014"
Sep 10 12:58:51.203051 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.203031    2577 scope.go:117] "RemoveContainer" containerID="42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014"
Sep 10 12:58:51.203302 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:58:51.203283    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014\": container with ID starting with 42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014 not found: ID does not exist" containerID="42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014"
Sep 10 12:58:51.203351 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.203309    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014"} err="failed to get container status \"42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014\": rpc error: code = NotFound desc = could not find container \"42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014\": container with ID starting with 42e8362eeb3752b06eb2c8c25167e6bb1ac50ad913e98cd489747a8a5d754014 not found: ID does not exist"
Sep 10 12:58:51.227197 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.227144    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"]
Sep 10 12:58:51.229684 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.229665    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/embedding-w4-1312ea-kserve-575b45bc75-r5x48"]
Sep 10 12:58:51.335440 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:51.335410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"]
Sep 10 12:58:51.336981 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:58:51.336958    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd118765_ffb5_423e_9202_d59d02339b4b.slice/crio-2d39f934f8bc292a50626e312f70284de259b1beb3fa6cbac0b94dec3d8f4fb5 WatchSource:0}: Error finding container 2d39f934f8bc292a50626e312f70284de259b1beb3fa6cbac0b94dec3d8f4fb5: Status 404 returned error can't find the container with id 2d39f934f8bc292a50626e312f70284de259b1beb3fa6cbac0b94dec3d8f4fb5
Sep 10 12:58:52.001561 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:52.001482    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="763ee82a-1a3c-4bd9-8315-61e7fc00d191" path="/var/lib/kubelet/pods/763ee82a-1a3c-4bd9-8315-61e7fc00d191/volumes"
Sep 10 12:58:52.201942 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:52.201902    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd118765-ffb5-423e-9202-d59d02339b4b" containerID="505cdb488f007a32630ddb51bd94eb56aa7cbdd5c933bb307fb1525ec95eae01" exitCode=0
Sep 10 12:58:52.202319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:52.201981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz" event={"ID":"cd118765-ffb5-423e-9202-d59d02339b4b","Type":"ContainerDied","Data":"505cdb488f007a32630ddb51bd94eb56aa7cbdd5c933bb307fb1525ec95eae01"}
Sep 10 12:58:52.202319 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:52.202028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz" event={"ID":"cd118765-ffb5-423e-9202-d59d02339b4b","Type":"ContainerStarted","Data":"2d39f934f8bc292a50626e312f70284de259b1beb3fa6cbac0b94dec3d8f4fb5"}
Sep 10 12:58:53.346126 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:53.346102    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"
Sep 10 12:58:53.453049 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:53.451271    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd118765-ffb5-423e-9202-d59d02339b4b-maas-api-service-ca\") pod \"cd118765-ffb5-423e-9202-d59d02339b4b\" (UID: \"cd118765-ffb5-423e-9202-d59d02339b4b\") "
Sep 10 12:58:53.453049 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:53.452245    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd118765-ffb5-423e-9202-d59d02339b4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd118765-ffb5-423e-9202-d59d02339b4b" (UID: "cd118765-ffb5-423e-9202-d59d02339b4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:53.553199 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:53.553172    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd118765-ffb5-423e-9202-d59d02339b4b-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:58:53.603257 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:53.603235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:53.704133 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:53.704102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:53.933272 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:53.933179    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:54.214287 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:54.214211    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"
Sep 10 12:58:54.214287 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:54.214239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz" event={"ID":"cd118765-ffb5-423e-9202-d59d02339b4b","Type":"ContainerDied","Data":"2d39f934f8bc292a50626e312f70284de259b1beb3fa6cbac0b94dec3d8f4fb5"}
Sep 10 12:58:54.214287 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:54.214286    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d39f934f8bc292a50626e312f70284de259b1beb3fa6cbac0b94dec3d8f4fb5"
Sep 10 12:58:59.222145 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.222115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:59.340434 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.340377    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"]
Sep 10 12:58:59.341004 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.340986    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd118765-ffb5-423e-9202-d59d02339b4b" containerName="revoke-keys"
Sep 10 12:58:59.341004 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.341007    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd118765-ffb5-423e-9202-d59d02339b4b" containerName="revoke-keys"
Sep 10 12:58:59.341125 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.341081    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd118765-ffb5-423e-9202-d59d02339b4b" containerName="revoke-keys"
Sep 10 12:58:59.344163 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.344144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"
Sep 10 12:58:59.347059 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.347035    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:58:59.357961 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.357795    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"]
Sep 10 12:58:59.410701 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.410670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c2da4-c616-43ce-9141-4b1ccf1038b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6\" (UID: \"a80c2da4-c616-43ce-9141-4b1ccf1038b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"
Sep 10 12:58:59.512293 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.512224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c2da4-c616-43ce-9141-4b1ccf1038b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6\" (UID: \"a80c2da4-c616-43ce-9141-4b1ccf1038b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"
Sep 10 12:58:59.512869 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.512846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c2da4-c616-43ce-9141-4b1ccf1038b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6\" (UID: \"a80c2da4-c616-43ce-9141-4b1ccf1038b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"
Sep 10 12:58:59.655413 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.655382    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"
Sep 10 12:58:59.716937 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.716905    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:58:59.938043 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:58:59.938011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"]
Sep 10 12:58:59.941286 ip-10-0-140-169 kubenswrapper[2577]: W0910 12:58:59.941256    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80c2da4_c616_43ce_9141_4b1ccf1038b2.slice/crio-17a00a8ef73f7e84331a8175808bbb9a49c4946bfec5d6f05a2d2a86dcc691e1 WatchSource:0}: Error finding container 17a00a8ef73f7e84331a8175808bbb9a49c4946bfec5d6f05a2d2a86dcc691e1: Status 404 returned error can't find the container with id 17a00a8ef73f7e84331a8175808bbb9a49c4946bfec5d6f05a2d2a86dcc691e1
Sep 10 12:59:00.256235 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:00.256145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6" event={"ID":"a80c2da4-c616-43ce-9141-4b1ccf1038b2","Type":"ContainerStarted","Data":"fb9d578b86551036da5da477a74d28bd71232e1890e685b28300ab847a9c4eb7"}
Sep 10 12:59:00.256235 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:00.256229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6" event={"ID":"a80c2da4-c616-43ce-9141-4b1ccf1038b2","Type":"ContainerStarted","Data":"17a00a8ef73f7e84331a8175808bbb9a49c4946bfec5d6f05a2d2a86dcc691e1"}
Sep 10 12:59:00.285458 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:00.285414    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6" podStartSLOduration=1.285400608 podStartE2EDuration="1.285400608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:59:00.283664044 +0000 UTC m=+1714.805833096" watchObservedRunningTime="2026-09-10 12:59:00.285400608 +0000 UTC m=+1714.807569660"
Sep 10 12:59:01.263820 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.263779    2577 generic.go:356] "Generic (PLEG): container finished" podID="a80c2da4-c616-43ce-9141-4b1ccf1038b2" containerID="fb9d578b86551036da5da477a74d28bd71232e1890e685b28300ab847a9c4eb7" exitCode=0
Sep 10 12:59:01.264268 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.263868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6" event={"ID":"a80c2da4-c616-43ce-9141-4b1ccf1038b2","Type":"ContainerDied","Data":"fb9d578b86551036da5da477a74d28bd71232e1890e685b28300ab847a9c4eb7"}
Sep 10 12:59:01.473656 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.473612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:01.539635 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.539581    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"]
Sep 10 12:59:01.540246 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.540215    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj" podUID="cef0f523-e3bf-4101-b1ac-92b1676d00aa" containerName="payload-processing" containerID="cri-o://2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5" gracePeriod=30
Sep 10 12:59:01.543336 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.543315    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"]
Sep 10 12:59:01.543527 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.543505    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs" podUID="3644e83e-5927-4c4f-82b8-d87ae6ccc56e" containerName="payload-pre-processing" containerID="cri-o://0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d" gracePeriod=30
Sep 10 12:59:01.614296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.614269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:01.800733 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.800709    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:59:01.818883 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.818861    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:59:01.832854 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.832821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:01.938278 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.938241    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78sl5\" (UniqueName: \"kubernetes.io/projected/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-kube-api-access-78sl5\") pod \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\" (UID: \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\") "
Sep 10 12:59:01.938448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.938294    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef0f523-e3bf-4101-b1ac-92b1676d00aa-plugins-config-volume\") pod \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\" (UID: \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\") "
Sep 10 12:59:01.938448 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.938434    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-plugins-config-volume\") pod \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\" (UID: \"3644e83e-5927-4c4f-82b8-d87ae6ccc56e\") "
Sep 10 12:59:01.938541 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.938466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89qdf\" (UniqueName: \"kubernetes.io/projected/cef0f523-e3bf-4101-b1ac-92b1676d00aa-kube-api-access-89qdf\") pod \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\" (UID: \"cef0f523-e3bf-4101-b1ac-92b1676d00aa\") "
Sep 10 12:59:01.938883 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.938847    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef0f523-e3bf-4101-b1ac-92b1676d00aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cef0f523-e3bf-4101-b1ac-92b1676d00aa" (UID: "cef0f523-e3bf-4101-b1ac-92b1676d00aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:59:01.939032 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.938898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3644e83e-5927-4c4f-82b8-d87ae6ccc56e" (UID: "3644e83e-5927-4c4f-82b8-d87ae6ccc56e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:59:01.940556 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.940531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-kube-api-access-78sl5" (OuterVolumeSpecName: "kube-api-access-78sl5") pod "3644e83e-5927-4c4f-82b8-d87ae6ccc56e" (UID: "3644e83e-5927-4c4f-82b8-d87ae6ccc56e"). InnerVolumeSpecName "kube-api-access-78sl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:01.940797 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.940781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef0f523-e3bf-4101-b1ac-92b1676d00aa-kube-api-access-89qdf" (OuterVolumeSpecName: "kube-api-access-89qdf") pod "cef0f523-e3bf-4101-b1ac-92b1676d00aa" (UID: "cef0f523-e3bf-4101-b1ac-92b1676d00aa"). InnerVolumeSpecName "kube-api-access-89qdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:01.946723 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:01.946700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:02.003250 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.003219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"]
Sep 10 12:59:02.006685 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.006652    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-719d8a-84zzz"]
Sep 10 12:59:02.040361 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.040327    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:02.040361 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.040355    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qdf\" (UniqueName: \"kubernetes.io/projected/cef0f523-e3bf-4101-b1ac-92b1676d00aa-kube-api-access-89qdf\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:02.040361 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.040366    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78sl5\" (UniqueName: \"kubernetes.io/projected/3644e83e-5927-4c4f-82b8-d87ae6ccc56e-kube-api-access-78sl5\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:02.040565 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.040374    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cef0f523-e3bf-4101-b1ac-92b1676d00aa-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:02.043377 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.043313    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:02.270423 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.270385    2577 generic.go:356] "Generic (PLEG): container finished" podID="3644e83e-5927-4c4f-82b8-d87ae6ccc56e" containerID="0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d" exitCode=0
Sep 10 12:59:02.270900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.270437    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"
Sep 10 12:59:02.270900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.270472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs" event={"ID":"3644e83e-5927-4c4f-82b8-d87ae6ccc56e","Type":"ContainerDied","Data":"0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d"}
Sep 10 12:59:02.270900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.270531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs" event={"ID":"3644e83e-5927-4c4f-82b8-d87ae6ccc56e","Type":"ContainerDied","Data":"32a5915a67ea0910124da26704b34a2cee4b77bc75b711be58f31eaaa78551a2"}
Sep 10 12:59:02.270900 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.270549    2577 scope.go:117] "RemoveContainer" containerID="0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d"
Sep 10 12:59:02.272023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.272004    2577 generic.go:356] "Generic (PLEG): container finished" podID="cef0f523-e3bf-4101-b1ac-92b1676d00aa" containerID="2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5" exitCode=0
Sep 10 12:59:02.272128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.272032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj" event={"ID":"cef0f523-e3bf-4101-b1ac-92b1676d00aa","Type":"ContainerDied","Data":"2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5"}
Sep 10 12:59:02.272128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.272067    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"
Sep 10 12:59:02.272128 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.272089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj" event={"ID":"cef0f523-e3bf-4101-b1ac-92b1676d00aa","Type":"ContainerDied","Data":"1f4775c40f9060ab63ad6f9a089f968e9a5045dafa45789c28bc3d768d41ce01"}
Sep 10 12:59:02.280356 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.280330    2577 scope.go:117] "RemoveContainer" containerID="0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d"
Sep 10 12:59:02.280747 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:59:02.280724    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d\": container with ID starting with 0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d not found: ID does not exist" containerID="0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d"
Sep 10 12:59:02.280839 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.280760    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d"} err="failed to get container status \"0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d\": rpc error: code = NotFound desc = could not find container \"0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d\": container with ID starting with 0edf010f10776901ec25bcfc2c65910294c75c8433b4a7595c0fc20ac586868d not found: ID does not exist"
Sep 10 12:59:02.280839 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.280782    2577 scope.go:117] "RemoveContainer" containerID="2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5"
Sep 10 12:59:02.291679 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.291615    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"]
Sep 10 12:59:02.297783 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.297716    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-719d8a-59c4954845-tgqlj"]
Sep 10 12:59:02.298016 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.297977    2577 scope.go:117] "RemoveContainer" containerID="2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5"
Sep 10 12:59:02.298391 ip-10-0-140-169 kubenswrapper[2577]: E0910 12:59:02.298367    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5\": container with ID starting with 2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5 not found: ID does not exist" containerID="2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5"
Sep 10 12:59:02.298487 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.298396    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5"} err="failed to get container status \"2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5\": rpc error: code = NotFound desc = could not find container \"2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5\": container with ID starting with 2ebdb4cf0398aa951f3e5640f17c7a1f55e89ca2f86a1a276984df0d6f7169e5 not found: ID does not exist"
Sep 10 12:59:02.311515 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.311484    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"]
Sep 10 12:59:02.316872 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.316846    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-719d8a-5fb7787cd-6kchs"]
Sep 10 12:59:02.378218 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.378187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:02.403086 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.403061    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"
Sep 10 12:59:02.544878 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.544821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c2da4-c616-43ce-9141-4b1ccf1038b2-maas-api-service-ca\") pod \"a80c2da4-c616-43ce-9141-4b1ccf1038b2\" (UID: \"a80c2da4-c616-43ce-9141-4b1ccf1038b2\") "
Sep 10 12:59:02.545219 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.545193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c2da4-c616-43ce-9141-4b1ccf1038b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a80c2da4-c616-43ce-9141-4b1ccf1038b2" (UID: "a80c2da4-c616-43ce-9141-4b1ccf1038b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:59:02.645779 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:02.645741    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c2da4-c616-43ce-9141-4b1ccf1038b2-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:03.282619 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:03.282567    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"
Sep 10 12:59:03.283046 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:03.282621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6" event={"ID":"a80c2da4-c616-43ce-9141-4b1ccf1038b2","Type":"ContainerDied","Data":"17a00a8ef73f7e84331a8175808bbb9a49c4946bfec5d6f05a2d2a86dcc691e1"}
Sep 10 12:59:03.283046 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:03.282953    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17a00a8ef73f7e84331a8175808bbb9a49c4946bfec5d6f05a2d2a86dcc691e1"
Sep 10 12:59:04.001253 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:04.001211    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3644e83e-5927-4c4f-82b8-d87ae6ccc56e" path="/var/lib/kubelet/pods/3644e83e-5927-4c4f-82b8-d87ae6ccc56e/volumes"
Sep 10 12:59:04.001574 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:04.001560    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd118765-ffb5-423e-9202-d59d02339b4b" path="/var/lib/kubelet/pods/cd118765-ffb5-423e-9202-d59d02339b4b/volumes"
Sep 10 12:59:04.001903 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:04.001891    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cef0f523-e3bf-4101-b1ac-92b1676d00aa" path="/var/lib/kubelet/pods/cef0f523-e3bf-4101-b1ac-92b1676d00aa/volumes"
Sep 10 12:59:04.243971 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:04.243938    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"]
Sep 10 12:59:04.554299 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:04.554267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:09.468168 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:09.468131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:09.568681 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:09.568649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:09.786715 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:09.786687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:13.280820 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:13.280788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:13.383030 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:13.383000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:14.122429 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:14.122398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:15.172233 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.172207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:15.216494 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.216464    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"]
Sep 10 12:59:15.216740 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.216715    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r" podUID="bbbc2b36-948b-468c-8629-89779664042f" containerName="payload-processing" containerID="cri-o://c7cf659c87e9a256dee21f95df5a2ac183813002aeb4349066eede67bd0c69b0" gracePeriod=30
Sep 10 12:59:15.234520 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.234491    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"]
Sep 10 12:59:15.234913 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.234876    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq" podUID="5ced0f7d-b78b-4158-98ed-7242539fe3d8" containerName="payload-pre-processing" containerID="cri-o://c4280ea389e159314cbf99d96349065a1e063c62c4af9cc42596aae83e22ba02" gracePeriod=30
Sep 10 12:59:15.275657 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.275628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:15.354110 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.354070    2577 generic.go:356] "Generic (PLEG): container finished" podID="bbbc2b36-948b-468c-8629-89779664042f" containerID="c7cf659c87e9a256dee21f95df5a2ac183813002aeb4349066eede67bd0c69b0" exitCode=0
Sep 10 12:59:15.354251 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.354175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r" event={"ID":"bbbc2b36-948b-468c-8629-89779664042f","Type":"ContainerDied","Data":"c7cf659c87e9a256dee21f95df5a2ac183813002aeb4349066eede67bd0c69b0"}
Sep 10 12:59:15.356028 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.356000    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ced0f7d-b78b-4158-98ed-7242539fe3d8" containerID="c4280ea389e159314cbf99d96349065a1e063c62c4af9cc42596aae83e22ba02" exitCode=0
Sep 10 12:59:15.356156 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.356062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq" event={"ID":"5ced0f7d-b78b-4158-98ed-7242539fe3d8","Type":"ContainerDied","Data":"c4280ea389e159314cbf99d96349065a1e063c62c4af9cc42596aae83e22ba02"}
Sep 10 12:59:15.374727 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.374695    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:15.473057 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.473030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:15.490048 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.490025    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"]
Sep 10 12:59:15.491953 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.491932    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:59:15.494023 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.494001    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30919a-vx4k6"]
Sep 10 12:59:15.504331 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.504314    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:59:15.556441 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.556394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:15.674396 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.674368    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced0f7d-b78b-4158-98ed-7242539fe3d8-plugins-config-volume\") pod \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\" (UID: \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\") "
Sep 10 12:59:15.674396 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.674404    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc2b36-948b-468c-8629-89779664042f-plugins-config-volume\") pod \"bbbc2b36-948b-468c-8629-89779664042f\" (UID: \"bbbc2b36-948b-468c-8629-89779664042f\") "
Sep 10 12:59:15.674710 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.674497    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnzv7\" (UniqueName: \"kubernetes.io/projected/bbbc2b36-948b-468c-8629-89779664042f-kube-api-access-mnzv7\") pod \"bbbc2b36-948b-468c-8629-89779664042f\" (UID: \"bbbc2b36-948b-468c-8629-89779664042f\") "
Sep 10 12:59:15.674710 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.674525    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf2zs\" (UniqueName: \"kubernetes.io/projected/5ced0f7d-b78b-4158-98ed-7242539fe3d8-kube-api-access-nf2zs\") pod \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\" (UID: \"5ced0f7d-b78b-4158-98ed-7242539fe3d8\") "
Sep 10 12:59:15.674845 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.674814    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ced0f7d-b78b-4158-98ed-7242539fe3d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ced0f7d-b78b-4158-98ed-7242539fe3d8" (UID: "5ced0f7d-b78b-4158-98ed-7242539fe3d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:59:15.674906 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.674884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbc2b36-948b-468c-8629-89779664042f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbbc2b36-948b-468c-8629-89779664042f" (UID: "bbbc2b36-948b-468c-8629-89779664042f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:59:15.674970 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.674957    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced0f7d-b78b-4158-98ed-7242539fe3d8-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.676696 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.676668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ced0f7d-b78b-4158-98ed-7242539fe3d8-kube-api-access-nf2zs" (OuterVolumeSpecName: "kube-api-access-nf2zs") pod "5ced0f7d-b78b-4158-98ed-7242539fe3d8" (UID: "5ced0f7d-b78b-4158-98ed-7242539fe3d8"). InnerVolumeSpecName "kube-api-access-nf2zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:15.676785 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.676701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbc2b36-948b-468c-8629-89779664042f-kube-api-access-mnzv7" (OuterVolumeSpecName: "kube-api-access-mnzv7") pod "bbbc2b36-948b-468c-8629-89779664042f" (UID: "bbbc2b36-948b-468c-8629-89779664042f"). InnerVolumeSpecName "kube-api-access-mnzv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:15.776296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.776273    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbc2b36-948b-468c-8629-89779664042f-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.776296 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.776295    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzv7\" (UniqueName: \"kubernetes.io/projected/bbbc2b36-948b-468c-8629-89779664042f-kube-api-access-mnzv7\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.776437 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.776303    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf2zs\" (UniqueName: \"kubernetes.io/projected/5ced0f7d-b78b-4158-98ed-7242539fe3d8-kube-api-access-nf2zs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.883805 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:15.883771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:16.007316 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.007280    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a80c2da4-c616-43ce-9141-4b1ccf1038b2" path="/var/lib/kubelet/pods/a80c2da4-c616-43ce-9141-4b1ccf1038b2/volumes"
Sep 10 12:59:16.363478 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.363448    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"
Sep 10 12:59:16.363964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.363447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r" event={"ID":"bbbc2b36-948b-468c-8629-89779664042f","Type":"ContainerDied","Data":"e841ed6675b84dc9410be88a0cd18a6c5df89f4edb643f271390db37f3ded94d"}
Sep 10 12:59:16.363964 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.363607    2577 scope.go:117] "RemoveContainer" containerID="c7cf659c87e9a256dee21f95df5a2ac183813002aeb4349066eede67bd0c69b0"
Sep 10 12:59:16.365138 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.364945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq" event={"ID":"5ced0f7d-b78b-4158-98ed-7242539fe3d8","Type":"ContainerDied","Data":"fe5cb9611e083e1a952361395805f53fcf4d2d5e58f80657aa6c93a63c0e116c"}
Sep 10 12:59:16.365138 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.365031    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"
Sep 10 12:59:16.372954 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.372934    2577 scope.go:117] "RemoveContainer" containerID="c4280ea389e159314cbf99d96349065a1e063c62c4af9cc42596aae83e22ba02"
Sep 10 12:59:16.404663 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.404636    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"]
Sep 10 12:59:16.410357 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.410335    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30919a-55b4759c65-bcd9r"]
Sep 10 12:59:16.450859 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.450834    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"]
Sep 10 12:59:16.458018 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:16.457996    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30919a-5f6b9896bb-94xlq"]
Sep 10 12:59:18.000509 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:18.000468    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ced0f7d-b78b-4158-98ed-7242539fe3d8" path="/var/lib/kubelet/pods/5ced0f7d-b78b-4158-98ed-7242539fe3d8/volumes"
Sep 10 12:59:18.001026 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:18.001011    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbbc2b36-948b-468c-8629-89779664042f" path="/var/lib/kubelet/pods/bbbc2b36-948b-468c-8629-89779664042f/volumes"
Sep 10 12:59:19.263474 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.263432    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm" podUID="f3add0af-726e-410b-91fc-f2d22ba198de" containerName="main" containerID="cri-o://aa36b6a717943e21e63af9dff3e9e1887affd1b138e9b7110fe99d33b4867cda" gracePeriod=45
Sep 10 12:59:19.384342 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.384310    2577 generic.go:356] "Generic (PLEG): container finished" podID="f3add0af-726e-410b-91fc-f2d22ba198de" containerID="aa36b6a717943e21e63af9dff3e9e1887affd1b138e9b7110fe99d33b4867cda" exitCode=0
Sep 10 12:59:19.384475 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.384385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm" event={"ID":"f3add0af-726e-410b-91fc-f2d22ba198de","Type":"ContainerDied","Data":"aa36b6a717943e21e63af9dff3e9e1887affd1b138e9b7110fe99d33b4867cda"}
Sep 10 12:59:19.510942 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.510920    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:59:19.611317 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhj5q\" (UniqueName: \"kubernetes.io/projected/f3add0af-726e-410b-91fc-f2d22ba198de-kube-api-access-rhj5q\") pod \"f3add0af-726e-410b-91fc-f2d22ba198de\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") "
Sep 10 12:59:19.611472 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-tmp-dir\") pod \"f3add0af-726e-410b-91fc-f2d22ba198de\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") "
Sep 10 12:59:19.611472 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-dshm\") pod \"f3add0af-726e-410b-91fc-f2d22ba198de\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") "
Sep 10 12:59:19.611472 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-home\") pod \"f3add0af-726e-410b-91fc-f2d22ba198de\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") "
Sep 10 12:59:19.611472 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-model-cache\") pod \"f3add0af-726e-410b-91fc-f2d22ba198de\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") "
Sep 10 12:59:19.611472 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611460    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3add0af-726e-410b-91fc-f2d22ba198de-tls-certs\") pod \"f3add0af-726e-410b-91fc-f2d22ba198de\" (UID: \"f3add0af-726e-410b-91fc-f2d22ba198de\") "
Sep 10 12:59:19.611777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3add0af-726e-410b-91fc-f2d22ba198de" (UID: "f3add0af-726e-410b-91fc-f2d22ba198de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.611777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611687    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-home" (OuterVolumeSpecName: "home") pod "f3add0af-726e-410b-91fc-f2d22ba198de" (UID: "f3add0af-726e-410b-91fc-f2d22ba198de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.611777 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3add0af-726e-410b-91fc-f2d22ba198de" (UID: "f3add0af-726e-410b-91fc-f2d22ba198de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.611932 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611850    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.611932 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611872    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.611932 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.611885    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.613505 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.613480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-dshm" (OuterVolumeSpecName: "dshm") pod "f3add0af-726e-410b-91fc-f2d22ba198de" (UID: "f3add0af-726e-410b-91fc-f2d22ba198de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.613633 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.613529    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3add0af-726e-410b-91fc-f2d22ba198de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3add0af-726e-410b-91fc-f2d22ba198de" (UID: "f3add0af-726e-410b-91fc-f2d22ba198de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:59:19.613633 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.613571    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3add0af-726e-410b-91fc-f2d22ba198de-kube-api-access-rhj5q" (OuterVolumeSpecName: "kube-api-access-rhj5q") pod "f3add0af-726e-410b-91fc-f2d22ba198de" (UID: "f3add0af-726e-410b-91fc-f2d22ba198de"). InnerVolumeSpecName "kube-api-access-rhj5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:19.712404 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.712378    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhj5q\" (UniqueName: \"kubernetes.io/projected/f3add0af-726e-410b-91fc-f2d22ba198de-kube-api-access-rhj5q\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.712404 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.712402    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3add0af-726e-410b-91fc-f2d22ba198de-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.712572 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:19.712413    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3add0af-726e-410b-91fc-f2d22ba198de-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 12:59:20.390584 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:20.390558    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"
Sep 10 12:59:20.391021 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:20.390616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm" event={"ID":"f3add0af-726e-410b-91fc-f2d22ba198de","Type":"ContainerDied","Data":"a3a8c43ca460131b00bf26b1fe5a499d135eccde5273cfe317871cf2b08940bf"}
Sep 10 12:59:20.391021 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:20.390675    2577 scope.go:117] "RemoveContainer" containerID="aa36b6a717943e21e63af9dff3e9e1887affd1b138e9b7110fe99d33b4867cda"
Sep 10 12:59:20.409879 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:20.409855    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"]
Sep 10 12:59:20.422159 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:20.422120    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/distinct-2-w0-30919a-kserve-76ff997f75-d2xdm"]
Sep 10 12:59:20.569990 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:20.569961    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:22.000664 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:22.000631    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3add0af-726e-410b-91fc-f2d22ba198de" path="/var/lib/kubelet/pods/f3add0af-726e-410b-91fc-f2d22ba198de/volumes"
Sep 10 12:59:23.419825 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:23.419767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:23.525824 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:23.525787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:23.794810 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:23.794777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:26.266985 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:26.266956    2577 scope.go:117] "RemoveContainer" containerID="63db7dc82af5086582c6b7d31d29e0811a270b7c6887a032a48119ca20f9fad4"
Sep 10 12:59:42.977388 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:42.977346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:43.062804 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:43.062769    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:43.168676 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:43.168646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:44.366263 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:44.366234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:50.398582 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:50.398535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:58.059993 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:58.059959    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"]
Sep 10 12:59:58.075742 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:58.075718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 12:59:59.028586 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:59.028548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"]
Sep 10 12:59:59.040784 ip-10-0-140-169 kubenswrapper[2577]: I0910 12:59:59.040754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:00:00.143555 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.143522    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"]
Sep 10 13:00:00.144171 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144148    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3644e83e-5927-4c4f-82b8-d87ae6ccc56e" containerName="payload-pre-processing"
Sep 10 13:00:00.144267 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144173    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3644e83e-5927-4c4f-82b8-d87ae6ccc56e" containerName="payload-pre-processing"
Sep 10 13:00:00.144267 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144186    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80c2da4-c616-43ce-9141-4b1ccf1038b2" containerName="revoke-keys"
Sep 10 13:00:00.144267 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144194    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80c2da4-c616-43ce-9141-4b1ccf1038b2" containerName="revoke-keys"
Sep 10 13:00:00.144267 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144210    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3add0af-726e-410b-91fc-f2d22ba198de" containerName="main"
Sep 10 13:00:00.144267 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144220    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3add0af-726e-410b-91fc-f2d22ba198de" containerName="main"
Sep 10 13:00:00.144267 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144248    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef0f523-e3bf-4101-b1ac-92b1676d00aa" containerName="payload-processing"
Sep 10 13:00:00.144267 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144257    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef0f523-e3bf-4101-b1ac-92b1676d00aa" containerName="payload-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144276    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbc2b36-948b-468c-8629-89779664042f" containerName="payload-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144285    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbc2b36-948b-468c-8629-89779664042f" containerName="payload-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144303    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ced0f7d-b78b-4158-98ed-7242539fe3d8" containerName="payload-pre-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144312    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ced0f7d-b78b-4158-98ed-7242539fe3d8" containerName="payload-pre-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144412    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ced0f7d-b78b-4158-98ed-7242539fe3d8" containerName="payload-pre-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144426    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a80c2da4-c616-43ce-9141-4b1ccf1038b2" containerName="revoke-keys"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144437    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3644e83e-5927-4c4f-82b8-d87ae6ccc56e" containerName="payload-pre-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144449    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbbc2b36-948b-468c-8629-89779664042f" containerName="payload-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef0f523-e3bf-4101-b1ac-92b1676d00aa" containerName="payload-processing"
Sep 10 13:00:00.144657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.144469    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3add0af-726e-410b-91fc-f2d22ba198de" containerName="main"
Sep 10 13:00:00.147703 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.147681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"
Sep 10 13:00:00.150586 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.150557    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"]
Sep 10 13:00:00.151513 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.151496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 13:00:00.153820 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.153803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"
Sep 10 13:00:00.158452 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.158435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"
Sep 10 13:00:00.161907 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.161888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"]
Sep 10 13:00:00.170829 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.170773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"]
Sep 10 13:00:00.172784 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.172763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"
Sep 10 13:00:00.451397 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.451369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"]
Sep 10 13:00:00.453315 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:00:00.453286    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb55da63_8b46_43fa_a7c1_6b5f034c6ef4.slice/crio-fe86a6aea378e71bf75b94df1a9e9efd9c9802b26e0dc76d9f6c7dcf6fdbdebe WatchSource:0}: Error finding container fe86a6aea378e71bf75b94df1a9e9efd9c9802b26e0dc76d9f6c7dcf6fdbdebe: Status 404 returned error can't find the container with id fe86a6aea378e71bf75b94df1a9e9efd9c9802b26e0dc76d9f6c7dcf6fdbdebe
Sep 10 13:00:00.465937 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.465912    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"]
Sep 10 13:00:00.468369 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:00:00.468342    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651aa5a6_0a33_471d_98fc_c3c40a994cd6.slice/crio-f8bb8a8744d03bdfbe83b0c3dd838c770f6e66bf643df5b9f424bfc0393bf15c WatchSource:0}: Error finding container f8bb8a8744d03bdfbe83b0c3dd838c770f6e66bf643df5b9f424bfc0393bf15c: Status 404 returned error can't find the container with id f8bb8a8744d03bdfbe83b0c3dd838c770f6e66bf643df5b9f424bfc0393bf15c
Sep 10 13:00:00.610823 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.610783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx" event={"ID":"fb55da63-8b46-43fa-a7c1-6b5f034c6ef4","Type":"ContainerStarted","Data":"fe86a6aea378e71bf75b94df1a9e9efd9c9802b26e0dc76d9f6c7dcf6fdbdebe"}
Sep 10 13:00:00.611944 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:00.611923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6" event={"ID":"651aa5a6-0a33-471d-98fc-c3c40a994cd6","Type":"ContainerStarted","Data":"f8bb8a8744d03bdfbe83b0c3dd838c770f6e66bf643df5b9f424bfc0393bf15c"}
Sep 10 13:00:01.618925 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:01.618828    2577 generic.go:356] "Generic (PLEG): container finished" podID="651aa5a6-0a33-471d-98fc-c3c40a994cd6" containerID="e0b1c77d0830b6afa8d1efcbbd1a114abc9f959d5559a1499583b3a7afa47f44" exitCode=0
Sep 10 13:00:01.618925 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:01.618888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6" event={"ID":"651aa5a6-0a33-471d-98fc-c3c40a994cd6","Type":"ContainerDied","Data":"e0b1c77d0830b6afa8d1efcbbd1a114abc9f959d5559a1499583b3a7afa47f44"}
Sep 10 13:00:01.620464 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:01.620443    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb55da63-8b46-43fa-a7c1-6b5f034c6ef4" containerID="c6aa70f901913d5c670461aeca7cbf0c43ff5c984611effda17e6e180f49e47d" exitCode=0
Sep 10 13:00:01.620621 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:01.620523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx" event={"ID":"fb55da63-8b46-43fa-a7c1-6b5f034c6ef4","Type":"ContainerDied","Data":"c6aa70f901913d5c670461aeca7cbf0c43ff5c984611effda17e6e180f49e47d"}
Sep 10 13:00:02.762675 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:02.762652    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"
Sep 10 13:00:02.798311 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:02.798285    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"
Sep 10 13:00:03.631906 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:03.631875    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"
Sep 10 13:00:03.632096 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:03.631907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx" event={"ID":"fb55da63-8b46-43fa-a7c1-6b5f034c6ef4","Type":"ContainerDied","Data":"fe86a6aea378e71bf75b94df1a9e9efd9c9802b26e0dc76d9f6c7dcf6fdbdebe"}
Sep 10 13:00:03.632096 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:03.631952    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe86a6aea378e71bf75b94df1a9e9efd9c9802b26e0dc76d9f6c7dcf6fdbdebe"
Sep 10 13:00:03.633283 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:03.633256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6" event={"ID":"651aa5a6-0a33-471d-98fc-c3c40a994cd6","Type":"ContainerDied","Data":"f8bb8a8744d03bdfbe83b0c3dd838c770f6e66bf643df5b9f424bfc0393bf15c"}
Sep 10 13:00:03.633283 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:03.633276    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"
Sep 10 13:00:03.633440 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:03.633291    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8bb8a8744d03bdfbe83b0c3dd838c770f6e66bf643df5b9f424bfc0393bf15c"
Sep 10 13:00:11.407906 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:11.407868    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:00:13.079458 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.079408    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv" podUID="4825c6d9-ce15-426d-a079-a0aa6b804823" containerName="main" containerID="cri-o://c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721" gracePeriod=45
Sep 10 13:00:13.328536 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.328512    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 13:00:13.379769 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.379689    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-home\") pod \"4825c6d9-ce15-426d-a079-a0aa6b804823\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") "
Sep 10 13:00:13.379769 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.379736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4825c6d9-ce15-426d-a079-a0aa6b804823-tls-certs\") pod \"4825c6d9-ce15-426d-a079-a0aa6b804823\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") "
Sep 10 13:00:13.379769 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.379770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-model-cache\") pod \"4825c6d9-ce15-426d-a079-a0aa6b804823\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") "
Sep 10 13:00:13.380033 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.379815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-dshm\") pod \"4825c6d9-ce15-426d-a079-a0aa6b804823\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") "
Sep 10 13:00:13.380033 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.379836    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-tmp-dir\") pod \"4825c6d9-ce15-426d-a079-a0aa6b804823\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") "
Sep 10 13:00:13.380033 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.379887    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqxbb\" (UniqueName: \"kubernetes.io/projected/4825c6d9-ce15-426d-a079-a0aa6b804823-kube-api-access-cqxbb\") pod \"4825c6d9-ce15-426d-a079-a0aa6b804823\" (UID: \"4825c6d9-ce15-426d-a079-a0aa6b804823\") "
Sep 10 13:00:13.380033 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.379978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-home" (OuterVolumeSpecName: "home") pod "4825c6d9-ce15-426d-a079-a0aa6b804823" (UID: "4825c6d9-ce15-426d-a079-a0aa6b804823"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:00:13.380304 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.380099    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-model-cache" (OuterVolumeSpecName: "model-cache") pod "4825c6d9-ce15-426d-a079-a0aa6b804823" (UID: "4825c6d9-ce15-426d-a079-a0aa6b804823"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:00:13.380365 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.380177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4825c6d9-ce15-426d-a079-a0aa6b804823" (UID: "4825c6d9-ce15-426d-a079-a0aa6b804823"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:00:13.380365 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.380185    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:13.382090 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.382057    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-dshm" (OuterVolumeSpecName: "dshm") pod "4825c6d9-ce15-426d-a079-a0aa6b804823" (UID: "4825c6d9-ce15-426d-a079-a0aa6b804823"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:00:13.382090 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.382079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4825c6d9-ce15-426d-a079-a0aa6b804823-kube-api-access-cqxbb" (OuterVolumeSpecName: "kube-api-access-cqxbb") pod "4825c6d9-ce15-426d-a079-a0aa6b804823" (UID: "4825c6d9-ce15-426d-a079-a0aa6b804823"). InnerVolumeSpecName "kube-api-access-cqxbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:00:13.382242 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.382151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4825c6d9-ce15-426d-a079-a0aa6b804823-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4825c6d9-ce15-426d-a079-a0aa6b804823" (UID: "4825c6d9-ce15-426d-a079-a0aa6b804823"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:00:13.480911 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.480886    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4825c6d9-ce15-426d-a079-a0aa6b804823-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:13.480911 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.480907    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:13.480911 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.480916    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:13.481092 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.480924    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825c6d9-ce15-426d-a079-a0aa6b804823-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:13.481092 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.480933    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxbb\" (UniqueName: \"kubernetes.io/projected/4825c6d9-ce15-426d-a079-a0aa6b804823-kube-api-access-cqxbb\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:13.692444 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.692362    2577 generic.go:356] "Generic (PLEG): container finished" podID="4825c6d9-ce15-426d-a079-a0aa6b804823" containerID="c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721" exitCode=0
Sep 10 13:00:13.692444 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.692405    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"
Sep 10 13:00:13.692714 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.692456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv" event={"ID":"4825c6d9-ce15-426d-a079-a0aa6b804823","Type":"ContainerDied","Data":"c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721"}
Sep 10 13:00:13.692714 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.692505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv" event={"ID":"4825c6d9-ce15-426d-a079-a0aa6b804823","Type":"ContainerDied","Data":"7393f5d49af0948fe1d4219e1d1161330cc94969c1869a883ffa5222fb5bfff1"}
Sep 10 13:00:13.692714 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.692532    2577 scope.go:117] "RemoveContainer" containerID="c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721"
Sep 10 13:00:13.704657 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.704637    2577 scope.go:117] "RemoveContainer" containerID="c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721"
Sep 10 13:00:13.705183 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:00:13.705164    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721\": container with ID starting with c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721 not found: ID does not exist" containerID="c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721"
Sep 10 13:00:13.705262 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.705192    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721"} err="failed to get container status \"c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721\": rpc error: code = NotFound desc = could not find container \"c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721\": container with ID starting with c0edb233d8221b9e83f0726d30bb70f5168d8bc339194771c5c4cad30faf2721 not found: ID does not exist"
Sep 10 13:00:13.726505 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.726482    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"]
Sep 10 13:00:13.733138 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:13.733119    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/auth-test-model-c5adae-kserve-7d84d5594c-lvccv"]
Sep 10 13:00:14.005619 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.005534    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4825c6d9-ce15-426d-a079-a0aa6b804823" path="/var/lib/kubelet/pods/4825c6d9-ce15-426d-a079-a0aa6b804823/volumes"
Sep 10 13:00:14.048217 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.048184    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj" podUID="5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" containerName="main" containerID="cri-o://cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050" gracePeriod=45
Sep 10 13:00:14.297273 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.297251    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 13:00:14.389245 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-model-cache\") pod \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") "
Sep 10 13:00:14.389390 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-home\") pod \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") "
Sep 10 13:00:14.389390 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tls-certs\") pod \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") "
Sep 10 13:00:14.389390 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tmp-dir\") pod \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") "
Sep 10 13:00:14.389508 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-dshm\") pod \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") "
Sep 10 13:00:14.389508 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9xqn\" (UniqueName: \"kubernetes.io/projected/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-kube-api-access-g9xqn\") pod \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\" (UID: \"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb\") "
Sep 10 13:00:14.389638 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389482    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" (UID: "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:00:14.389638 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389554    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-home" (OuterVolumeSpecName: "home") pod "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" (UID: "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:00:14.389755 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389652    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" (UID: "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:00:14.389817 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389752    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tmp-dir\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:14.389817 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389767    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-model-cache\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:14.389817 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.389776    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-home\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:14.391457 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.391423    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-dshm" (OuterVolumeSpecName: "dshm") pod "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" (UID: "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:00:14.391553 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.391473    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" (UID: "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:00:14.391553 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.391482    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-kube-api-access-g9xqn" (OuterVolumeSpecName: "kube-api-access-g9xqn") pod "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" (UID: "5f47cbc6-80d8-4e6a-a1e5-021c77454bbb"). InnerVolumeSpecName "kube-api-access-g9xqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:00:14.491177 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.491132    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-tls-certs\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:14.491177 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.491171    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-dshm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:14.491177 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.491181    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xqn\" (UniqueName: \"kubernetes.io/projected/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb-kube-api-access-g9xqn\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:00:14.699122 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.699030    2577 generic.go:356] "Generic (PLEG): container finished" podID="5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" containerID="cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050" exitCode=0
Sep 10 13:00:14.699122 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.699074    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"
Sep 10 13:00:14.699342 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.699121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj" event={"ID":"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb","Type":"ContainerDied","Data":"cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050"}
Sep 10 13:00:14.699342 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.699167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj" event={"ID":"5f47cbc6-80d8-4e6a-a1e5-021c77454bbb","Type":"ContainerDied","Data":"6a0ca12c7edfb0ae58ece9cae277b2e75aa4cf9dbedc07e4ee6d5ec1951b7bc3"}
Sep 10 13:00:14.699342 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.699184    2577 scope.go:117] "RemoveContainer" containerID="cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050"
Sep 10 13:00:14.709744 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.709721    2577 scope.go:117] "RemoveContainer" containerID="cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050"
Sep 10 13:00:14.710004 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:00:14.709985    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050\": container with ID starting with cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050 not found: ID does not exist" containerID="cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050"
Sep 10 13:00:14.710049 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.710014    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050"} err="failed to get container status \"cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050\": rpc error: code = NotFound desc = could not find container \"cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050\": container with ID starting with cee4568457a8f72d5bf28fe1d3976ea5031486e5c0ce2045aba24092aeed0050 not found: ID does not exist"
Sep 10 13:00:14.742541 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.742514    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"]
Sep 10 13:00:14.751044 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:14.751022    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/auth-test-model-e5208b-kserve-7f5769c5ff-55hxj"]
Sep 10 13:00:16.000569 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:16.000527    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" path="/var/lib/kubelet/pods/5f47cbc6-80d8-4e6a-a1e5-021c77454bbb/volumes"
Sep 10 13:00:26.054437 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:26.054401    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:00:26.059111 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:26.059087    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 13:00:26.061660 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:26.061639    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:00:26.066426 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:26.066402    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 13:00:26.332123 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:26.332048    2577 scope.go:117] "RemoveContainer" containerID="71f23648012d6daace2ba21a6e5298b3f27eeb1e66bdea49b91134a4dc733b37"
Sep 10 13:00:54.319860 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:00:54.319825    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:01:08.953924 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:01:08.953886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:01:38.224985 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:01:38.224946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:01:54.421300 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:01:54.421219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:01:55.289330 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:01:55.289297    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:02:00.625659 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:02:00.625589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:02:00.771120 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:02:00.771072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:02:06.995430 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:02:06.995376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:02:25.556523 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:02:25.556478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:02:32.415368 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:02:32.415309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:11.778644 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:11.778607    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:12.883705 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:12.883623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:13.280072 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:13.280044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:13.722344 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:13.722216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:13.866951 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:13.866907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:16.743136 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:16.743092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:16.935380 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:16.935333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:17.063952 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:17.063917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:18.804427 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:18.804346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:18.965685 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:18.965655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:20.154821 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.154789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:20.356761 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.356729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:20.717502 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717469    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"]
Sep 10 13:03:20.717920 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717908    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" containerName="main"
Sep 10 13:03:20.717969 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717922    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" containerName="main"
Sep 10 13:03:20.717969 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717945    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb55da63-8b46-43fa-a7c1-6b5f034c6ef4" containerName="cleanup"
Sep 10 13:03:20.717969 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717950    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb55da63-8b46-43fa-a7c1-6b5f034c6ef4" containerName="cleanup"
Sep 10 13:03:20.717969 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717961    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4825c6d9-ce15-426d-a079-a0aa6b804823" containerName="main"
Sep 10 13:03:20.717969 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717966    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4825c6d9-ce15-426d-a079-a0aa6b804823" containerName="main"
Sep 10 13:03:20.718129 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717972    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="651aa5a6-0a33-471d-98fc-c3c40a994cd6" containerName="cleanup"
Sep 10 13:03:20.718129 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.717978    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="651aa5a6-0a33-471d-98fc-c3c40a994cd6" containerName="cleanup"
Sep 10 13:03:20.718129 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.718031    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4825c6d9-ce15-426d-a079-a0aa6b804823" containerName="main"
Sep 10 13:03:20.718129 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.718037    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f47cbc6-80d8-4e6a-a1e5-021c77454bbb" containerName="main"
Sep 10 13:03:20.718129 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.718045    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="651aa5a6-0a33-471d-98fc-c3c40a994cd6" containerName="cleanup"
Sep 10 13:03:20.718129 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.718050    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb55da63-8b46-43fa-a7c1-6b5f034c6ef4" containerName="cleanup"
Sep 10 13:03:20.721243 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.721225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:20.732982 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.732963    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c91a45-dockercfg-cgkzm\""
Sep 10 13:03:20.736357 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.736338    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c91a45\""
Sep 10 13:03:20.765122 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.765096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"]
Sep 10 13:03:20.776363 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.776330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95\" (UID: \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:20.776471 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.776420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4cmg\" (UniqueName: \"kubernetes.io/projected/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-kube-api-access-j4cmg\") pod \"payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95\" (UID: \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:20.854614 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.854565    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"]
Sep 10 13:03:20.858168 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.858152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:20.874711 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.874686    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"]
Sep 10 13:03:20.877034 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.877004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95\" (UID: \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:20.877119 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.877082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4cmg\" (UniqueName: \"kubernetes.io/projected/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-kube-api-access-j4cmg\") pod \"payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95\" (UID: \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:20.877984 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.877957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95\" (UID: \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:20.895296 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.895263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4cmg\" (UniqueName: \"kubernetes.io/projected/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-kube-api-access-j4cmg\") pod \"payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95\" (UID: \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:20.978741 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.978643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j628j\" (UniqueName: \"kubernetes.io/projected/aee60f3a-a8a1-4338-924c-5a8765de78f5-kube-api-access-j628j\") pod \"payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz\" (UID: \"aee60f3a-a8a1-4338-924c-5a8765de78f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:20.978741 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:20.978704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee60f3a-a8a1-4338-924c-5a8765de78f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz\" (UID: \"aee60f3a-a8a1-4338-924c-5a8765de78f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:21.031773 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.031727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:21.080375 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.079642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j628j\" (UniqueName: \"kubernetes.io/projected/aee60f3a-a8a1-4338-924c-5a8765de78f5-kube-api-access-j628j\") pod \"payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz\" (UID: \"aee60f3a-a8a1-4338-924c-5a8765de78f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:21.080375 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.079786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee60f3a-a8a1-4338-924c-5a8765de78f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz\" (UID: \"aee60f3a-a8a1-4338-924c-5a8765de78f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:21.081034 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.080860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee60f3a-a8a1-4338-924c-5a8765de78f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz\" (UID: \"aee60f3a-a8a1-4338-924c-5a8765de78f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:21.109081 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.109052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j628j\" (UniqueName: \"kubernetes.io/projected/aee60f3a-a8a1-4338-924c-5a8765de78f5-kube-api-access-j628j\") pod \"payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz\" (UID: \"aee60f3a-a8a1-4338-924c-5a8765de78f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:21.169197 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.169168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:21.344660 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.344636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"]
Sep 10 13:03:21.346800 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:03:21.346770    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3858d0a_fb6b_4021_8fb0_0a2eb8a60efc.slice/crio-2aa2ff4fba4eb7b27061a58779de81429932b5b0dc465dc70b94cf6b040cb03d WatchSource:0}: Error finding container 2aa2ff4fba4eb7b27061a58779de81429932b5b0dc465dc70b94cf6b040cb03d: Status 404 returned error can't find the container with id 2aa2ff4fba4eb7b27061a58779de81429932b5b0dc465dc70b94cf6b040cb03d
Sep 10 13:03:21.498823 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.498790    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"]
Sep 10 13:03:21.502587 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:03:21.502559    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee60f3a_a8a1_4338_924c_5a8765de78f5.slice/crio-76f120b9ff7edec843f473326cf1c708ed05d69ca0445a6bb46767006c3b771f WatchSource:0}: Error finding container 76f120b9ff7edec843f473326cf1c708ed05d69ca0445a6bb46767006c3b771f: Status 404 returned error can't find the container with id 76f120b9ff7edec843f473326cf1c708ed05d69ca0445a6bb46767006c3b771f
Sep 10 13:03:21.689167 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.689069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz" event={"ID":"aee60f3a-a8a1-4338-924c-5a8765de78f5","Type":"ContainerStarted","Data":"2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3"}
Sep 10 13:03:21.689319 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.689221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:21.689319 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.689271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz" event={"ID":"aee60f3a-a8a1-4338-924c-5a8765de78f5","Type":"ContainerStarted","Data":"76f120b9ff7edec843f473326cf1c708ed05d69ca0445a6bb46767006c3b771f"}
Sep 10 13:03:21.690664 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.690631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95" event={"ID":"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc","Type":"ContainerStarted","Data":"75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8"}
Sep 10 13:03:21.690794 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.690675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95" event={"ID":"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc","Type":"ContainerStarted","Data":"2aa2ff4fba4eb7b27061a58779de81429932b5b0dc465dc70b94cf6b040cb03d"}
Sep 10 13:03:21.690794 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.690697    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:21.784743 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.784682    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz" podStartSLOduration=1.784660869 podStartE2EDuration="1.784660869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:03:21.784032478 +0000 UTC m=+1976.306201529" watchObservedRunningTime="2026-09-10 13:03:21.784660869 +0000 UTC m=+1976.306829942"
Sep 10 13:03:21.869329 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:21.869266    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95" podStartSLOduration=1.869244799 podStartE2EDuration="1.869244799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:03:21.866305073 +0000 UTC m=+1976.388474124" watchObservedRunningTime="2026-09-10 13:03:21.869244799 +0000 UTC m=+1976.391413856"
Sep 10 13:03:32.699257 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:32.699228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:03:32.699715 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:32.699293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:03:47.053429 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:47.053390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:47.622313 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:47.622276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:47.988352 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:47.988271    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:48.667986 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:48.667953    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:51.346423 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:51.346388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:51.435839 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:51.435801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:52.930821 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:52.930784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:53.074733 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:53.074704    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:53.185617 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:53.185527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:54.113413 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.113382    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:54.208203 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.208172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:03:54.259238 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.259198    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"]
Sep 10 13:03:54.263167 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.263147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:54.268092 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.268067    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6ea780\""
Sep 10 13:03:54.281136 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.281111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n\" (UID: \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:54.281268 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.281247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwrw\" (UniqueName: \"kubernetes.io/projected/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-kube-api-access-8qwrw\") pod \"payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n\" (UID: \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:54.291744 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.291717    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"]
Sep 10 13:03:54.295287 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.295262    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"]
Sep 10 13:03:54.299021 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.298999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:54.301791 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.301769    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6ea780-dockercfg-xqpgr\""
Sep 10 13:03:54.315632 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.315604    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"]
Sep 10 13:03:54.382205 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.382117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vccwz\" (UniqueName: \"kubernetes.io/projected/78d5215d-ce6f-4593-9c89-511d23946685-kube-api-access-vccwz\") pod \"payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8\" (UID: \"78d5215d-ce6f-4593-9c89-511d23946685\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:54.382205 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.382168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5215d-ce6f-4593-9c89-511d23946685-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8\" (UID: \"78d5215d-ce6f-4593-9c89-511d23946685\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:54.382388 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.382228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwrw\" (UniqueName: \"kubernetes.io/projected/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-kube-api-access-8qwrw\") pod \"payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n\" (UID: \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:54.382388 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.382352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n\" (UID: \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:54.383068 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.383049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n\" (UID: \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:54.394723 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.394696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwrw\" (UniqueName: \"kubernetes.io/projected/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-kube-api-access-8qwrw\") pod \"payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n\" (UID: \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:54.483187 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.483148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5215d-ce6f-4593-9c89-511d23946685-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8\" (UID: \"78d5215d-ce6f-4593-9c89-511d23946685\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:54.483354 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.483273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vccwz\" (UniqueName: \"kubernetes.io/projected/78d5215d-ce6f-4593-9c89-511d23946685-kube-api-access-vccwz\") pod \"payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8\" (UID: \"78d5215d-ce6f-4593-9c89-511d23946685\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:54.483862 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.483842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5215d-ce6f-4593-9c89-511d23946685-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8\" (UID: \"78d5215d-ce6f-4593-9c89-511d23946685\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:54.496267 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.496244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vccwz\" (UniqueName: \"kubernetes.io/projected/78d5215d-ce6f-4593-9c89-511d23946685-kube-api-access-vccwz\") pod \"payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8\" (UID: \"78d5215d-ce6f-4593-9c89-511d23946685\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:54.577642 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.577576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:54.610049 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.610014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:54.861701 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.861668    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"]
Sep 10 13:03:54.863189 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:03:54.863159    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693f8d1c_ebb8_4b8d_ab38_08e72a634b7c.slice/crio-dc2f08cd3e7419bac512061b1714f6138342531ad6486ffef68d311fc7a80031 WatchSource:0}: Error finding container dc2f08cd3e7419bac512061b1714f6138342531ad6486ffef68d311fc7a80031: Status 404 returned error can't find the container with id dc2f08cd3e7419bac512061b1714f6138342531ad6486ffef68d311fc7a80031
Sep 10 13:03:54.869959 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.869931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n" event={"ID":"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c","Type":"ContainerStarted","Data":"dc2f08cd3e7419bac512061b1714f6138342531ad6486ffef68d311fc7a80031"}
Sep 10 13:03:54.894708 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:54.894685    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"]
Sep 10 13:03:54.896729 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:03:54.896690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d5215d_ce6f_4593_9c89_511d23946685.slice/crio-3beb3b0da70b5116fb0a4903c920c4f2008606d95c5f96ddc49b9266681844c1 WatchSource:0}: Error finding container 3beb3b0da70b5116fb0a4903c920c4f2008606d95c5f96ddc49b9266681844c1: Status 404 returned error can't find the container with id 3beb3b0da70b5116fb0a4903c920c4f2008606d95c5f96ddc49b9266681844c1
Sep 10 13:03:55.876879 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:55.876841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n" event={"ID":"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c","Type":"ContainerStarted","Data":"f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8"}
Sep 10 13:03:55.877334 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:55.876928    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:03:55.878648 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:55.878625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8" event={"ID":"78d5215d-ce6f-4593-9c89-511d23946685","Type":"ContainerStarted","Data":"bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a"}
Sep 10 13:03:55.878770 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:55.878665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8" event={"ID":"78d5215d-ce6f-4593-9c89-511d23946685","Type":"ContainerStarted","Data":"3beb3b0da70b5116fb0a4903c920c4f2008606d95c5f96ddc49b9266681844c1"}
Sep 10 13:03:55.878770 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:55.878694    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:03:55.898500 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:55.898454    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n" podStartSLOduration=1.898440023 podStartE2EDuration="1.898440023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:03:55.89767318 +0000 UTC m=+2010.419842237" watchObservedRunningTime="2026-09-10 13:03:55.898440023 +0000 UTC m=+2010.420609074"
Sep 10 13:03:55.918670 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:03:55.918627    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8" podStartSLOduration=1.918589675 podStartE2EDuration="1.918589675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:03:55.917499476 +0000 UTC m=+2010.439668529" watchObservedRunningTime="2026-09-10 13:03:55.918589675 +0000 UTC m=+2010.440758726"
Sep 10 13:04:06.887255 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:06.887223    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:04:06.887671 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:06.887303    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:04:16.372702 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:16.372667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:04:21.808057 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:21.808027    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zc97k_68357e6d-f22c-40cf-b4b4-67bd383505e9/payload-processing/0.log"
Sep 10 13:04:21.933020 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:21.932993    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n_693f8d1c-ebb8-4b8d-ab38-08e72a634b7c/payload-processing/0.log"
Sep 10 13:04:26.385523 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:26.385488    2577 scope.go:117] "RemoveContainer" containerID="b213c7039796ff95c49554b102a16ef2f827ffd2dd9aa764bb4e515d1b939aaf"
Sep 10 13:04:26.394465 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:26.394445    2577 scope.go:117] "RemoveContainer" containerID="e7e8d3c4eea4116f40f6b728eb8b59616914f25e479feb9d742d8a349627b823"
Sep 10 13:04:30.079586 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:30.079544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:04:45.907153 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:45.907118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:04:55.005509 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:55.005422    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz_aee60f3a-a8a1-4338-924c-5a8765de78f5/payload-processing/0.log"
Sep 10 13:04:55.172154 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:55.172126    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zc97k_68357e6d-f22c-40cf-b4b4-67bd383505e9/payload-processing/0.log"
Sep 10 13:04:55.326604 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:55.326577    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n_693f8d1c-ebb8-4b8d-ab38-08e72a634b7c/payload-processing/0.log"
Sep 10 13:04:55.649224 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:55.649138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:04:56.223389 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:56.223351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:04:56.393255 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:56.393227    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:04:59.857803 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:04:59.857753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:00.019836 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:00.019799    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:00.207555 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:00.207481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:01.756972 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:01.756943    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:02.170745 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:02.170703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:02.684896 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:02.684861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:04.502541 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:04.502508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:04.793762 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:04.793727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:05.033907 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.033874    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"]
Sep 10 13:05:05.037205 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.037182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:05.049712 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.049691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c8b83b-dockercfg-lw6x2\""
Sep 10 13:05:05.050224 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.050207    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c8b83b\""
Sep 10 13:05:05.060686 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.060659    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"]
Sep 10 13:05:05.064120 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.064105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:05.074827 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.074805    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"]
Sep 10 13:05:05.115902 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.115862    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"]
Sep 10 13:05:05.134478 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.134446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz2fv\" (UniqueName: \"kubernetes.io/projected/3e7075cf-07bf-45b8-9038-de3a36389db0-kube-api-access-tz2fv\") pod \"payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq\" (UID: \"3e7075cf-07bf-45b8-9038-de3a36389db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:05.134678 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.134525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7075cf-07bf-45b8-9038-de3a36389db0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq\" (UID: \"3e7075cf-07bf-45b8-9038-de3a36389db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:05.235839 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.235800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9192d4a4-cf79-412d-b191-0422220c3b31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt\" (UID: \"9192d4a4-cf79-412d-b191-0422220c3b31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:05.236005 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.235883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz2fv\" (UniqueName: \"kubernetes.io/projected/3e7075cf-07bf-45b8-9038-de3a36389db0-kube-api-access-tz2fv\") pod \"payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq\" (UID: \"3e7075cf-07bf-45b8-9038-de3a36389db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:05.236005 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.235919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7075cf-07bf-45b8-9038-de3a36389db0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq\" (UID: \"3e7075cf-07bf-45b8-9038-de3a36389db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:05.236085 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.236007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcpm\" (UniqueName: \"kubernetes.io/projected/9192d4a4-cf79-412d-b191-0422220c3b31-kube-api-access-qzcpm\") pod \"payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt\" (UID: \"9192d4a4-cf79-412d-b191-0422220c3b31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:05.236681 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.236655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7075cf-07bf-45b8-9038-de3a36389db0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq\" (UID: \"3e7075cf-07bf-45b8-9038-de3a36389db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:05.251105 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.251076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz2fv\" (UniqueName: \"kubernetes.io/projected/3e7075cf-07bf-45b8-9038-de3a36389db0-kube-api-access-tz2fv\") pod \"payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq\" (UID: \"3e7075cf-07bf-45b8-9038-de3a36389db0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:05.337194 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.337104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcpm\" (UniqueName: \"kubernetes.io/projected/9192d4a4-cf79-412d-b191-0422220c3b31-kube-api-access-qzcpm\") pod \"payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt\" (UID: \"9192d4a4-cf79-412d-b191-0422220c3b31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:05.337194 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.337157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9192d4a4-cf79-412d-b191-0422220c3b31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt\" (UID: \"9192d4a4-cf79-412d-b191-0422220c3b31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:05.337867 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.337849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9192d4a4-cf79-412d-b191-0422220c3b31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt\" (UID: \"9192d4a4-cf79-412d-b191-0422220c3b31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:05.348832 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.348788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:05.353372 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.352118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcpm\" (UniqueName: \"kubernetes.io/projected/9192d4a4-cf79-412d-b191-0422220c3b31-kube-api-access-qzcpm\") pod \"payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt\" (UID: \"9192d4a4-cf79-412d-b191-0422220c3b31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:05.373920 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.373887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:05.673377 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.673344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"]
Sep 10 13:05:05.673794 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:05:05.673674    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7075cf_07bf_45b8_9038_de3a36389db0.slice/crio-0fa155735ff08f571efd57bd2688a14c45baf221e3f10edb43e629966084a849 WatchSource:0}: Error finding container 0fa155735ff08f571efd57bd2688a14c45baf221e3f10edb43e629966084a849: Status 404 returned error can't find the container with id 0fa155735ff08f571efd57bd2688a14c45baf221e3f10edb43e629966084a849
Sep 10 13:05:05.718995 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:05.718967    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"]
Sep 10 13:05:05.730740 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:05:05.730714    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9192d4a4_cf79_412d_b191_0422220c3b31.slice/crio-aacf5c0e22918e0c83ba10dfe8029e1438716c0c12e69a0373466e388552ab0c WatchSource:0}: Error finding container aacf5c0e22918e0c83ba10dfe8029e1438716c0c12e69a0373466e388552ab0c: Status 404 returned error can't find the container with id aacf5c0e22918e0c83ba10dfe8029e1438716c0c12e69a0373466e388552ab0c
Sep 10 13:05:06.260522 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:06.260480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt" event={"ID":"9192d4a4-cf79-412d-b191-0422220c3b31","Type":"ContainerStarted","Data":"30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7"}
Sep 10 13:05:06.260746 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:06.260583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:06.260807 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:06.260749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt" event={"ID":"9192d4a4-cf79-412d-b191-0422220c3b31","Type":"ContainerStarted","Data":"aacf5c0e22918e0c83ba10dfe8029e1438716c0c12e69a0373466e388552ab0c"}
Sep 10 13:05:06.262412 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:06.262382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq" event={"ID":"3e7075cf-07bf-45b8-9038-de3a36389db0","Type":"ContainerStarted","Data":"cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde"}
Sep 10 13:05:06.262542 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:06.262433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq" event={"ID":"3e7075cf-07bf-45b8-9038-de3a36389db0","Type":"ContainerStarted","Data":"0fa155735ff08f571efd57bd2688a14c45baf221e3f10edb43e629966084a849"}
Sep 10 13:05:06.262748 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:06.262722    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:06.309293 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:06.309245    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt" podStartSLOduration=2.30922997 podStartE2EDuration="2.30922997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:05:06.307744562 +0000 UTC m=+2080.829913642" watchObservedRunningTime="2026-09-10 13:05:06.30922997 +0000 UTC m=+2080.831399383"
Sep 10 13:05:06.370390 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:06.370347    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq" podStartSLOduration=2.370332312 podStartE2EDuration="2.370332312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:05:06.370192571 +0000 UTC m=+2080.892361625" watchObservedRunningTime="2026-09-10 13:05:06.370332312 +0000 UTC m=+2080.892501377"
Sep 10 13:05:17.271586 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:17.271553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:17.272072 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:17.271664    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:26.089769 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:26.089738    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:05:26.094418 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:26.094388    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 13:05:26.098171 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:26.098150    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:05:26.102613 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:26.102576    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 13:05:26.417771 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:26.417683    2577 scope.go:117] "RemoveContainer" containerID="fb9d578b86551036da5da477a74d28bd71232e1890e685b28300ab847a9c4eb7"
Sep 10 13:05:26.426692 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:26.426659    2577 scope.go:117] "RemoveContainer" containerID="505cdb488f007a32630ddb51bd94eb56aa7cbdd5c933bb307fb1525ec95eae01"
Sep 10 13:05:26.434535 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:26.434509    2577 scope.go:117] "RemoveContainer" containerID="807699fddd1f9e9acb4b1e8664f561b84b09aa161f0f273ed620c832444487b9"
Sep 10 13:05:29.989363 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:29.989309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:30.723231 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:30.723195    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"]
Sep 10 13:05:30.726763 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:30.726748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"
Sep 10 13:05:30.729678 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:30.729653    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 13:05:30.739795 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:30.739765    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"]
Sep 10 13:05:30.781020 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:30.780994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37bf498-7788-4d04-9c0b-f78c2de6162a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm\" (UID: \"f37bf498-7788-4d04-9c0b-f78c2de6162a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"
Sep 10 13:05:30.882373 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:30.882343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37bf498-7788-4d04-9c0b-f78c2de6162a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm\" (UID: \"f37bf498-7788-4d04-9c0b-f78c2de6162a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"
Sep 10 13:05:30.882954 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:30.882934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37bf498-7788-4d04-9c0b-f78c2de6162a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm\" (UID: \"f37bf498-7788-4d04-9c0b-f78c2de6162a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"
Sep 10 13:05:31.037102 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:31.037073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"
Sep 10 13:05:31.316451 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:31.316422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"]
Sep 10 13:05:31.317913 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:05:31.317885    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37bf498_7788_4d04_9c0b_f78c2de6162a.slice/crio-dc2035c06609ae2d9506c540bf9018866fa4081f884e6980fd5599f8c90f2087 WatchSource:0}: Error finding container dc2035c06609ae2d9506c540bf9018866fa4081f884e6980fd5599f8c90f2087: Status 404 returned error can't find the container with id dc2035c06609ae2d9506c540bf9018866fa4081f884e6980fd5599f8c90f2087
Sep 10 13:05:31.398347 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:31.398318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm" event={"ID":"f37bf498-7788-4d04-9c0b-f78c2de6162a","Type":"ContainerStarted","Data":"dc2035c06609ae2d9506c540bf9018866fa4081f884e6980fd5599f8c90f2087"}
Sep 10 13:05:32.405661 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:32.405547    2577 generic.go:356] "Generic (PLEG): container finished" podID="f37bf498-7788-4d04-9c0b-f78c2de6162a" containerID="bb5246be5fb3166d04ffd52aa715d530f605b927a4355f8a006a5af3b46874bd" exitCode=0
Sep 10 13:05:32.405661 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:32.405633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm" event={"ID":"f37bf498-7788-4d04-9c0b-f78c2de6162a","Type":"ContainerDied","Data":"bb5246be5fb3166d04ffd52aa715d530f605b927a4355f8a006a5af3b46874bd"}
Sep 10 13:05:33.545160 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:33.545136    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"
Sep 10 13:05:33.606390 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:33.606357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37bf498-7788-4d04-9c0b-f78c2de6162a-maas-api-service-ca\") pod \"f37bf498-7788-4d04-9c0b-f78c2de6162a\" (UID: \"f37bf498-7788-4d04-9c0b-f78c2de6162a\") "
Sep 10 13:05:33.606698 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:33.606676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37bf498-7788-4d04-9c0b-f78c2de6162a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f37bf498-7788-4d04-9c0b-f78c2de6162a" (UID: "f37bf498-7788-4d04-9c0b-f78c2de6162a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:05:33.708000 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:33.707923    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f37bf498-7788-4d04-9c0b-f78c2de6162a-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:05:34.418448 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:34.418344    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"
Sep 10 13:05:34.418448 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:34.418345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm" event={"ID":"f37bf498-7788-4d04-9c0b-f78c2de6162a","Type":"ContainerDied","Data":"dc2035c06609ae2d9506c540bf9018866fa4081f884e6980fd5599f8c90f2087"}
Sep 10 13:05:34.437360 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:34.418481    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2035c06609ae2d9506c540bf9018866fa4081f884e6980fd5599f8c90f2087"
Sep 10 13:05:43.960437 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:43.960398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:44.101501 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.101472    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"]
Sep 10 13:05:44.101711 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.101689    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq" podUID="3e7075cf-07bf-45b8-9038-de3a36389db0" containerName="payload-processing" containerID="cri-o://cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde" gracePeriod=30
Sep 10 13:05:44.129368 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.129341    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"]
Sep 10 13:05:44.129562 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.129542    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt" podUID="9192d4a4-cf79-412d-b191-0422220c3b31" containerName="payload-pre-processing" containerID="cri-o://30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7" gracePeriod=30
Sep 10 13:05:44.185828 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.185798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:44.322292 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.322192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:44.405620 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.403518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:44.409523 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.409501    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:44.473493 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.473461    2577 generic.go:356] "Generic (PLEG): container finished" podID="9192d4a4-cf79-412d-b191-0422220c3b31" containerID="30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7" exitCode=0
Sep 10 13:05:44.473710 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.473513    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"
Sep 10 13:05:44.473710 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.473555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt" event={"ID":"9192d4a4-cf79-412d-b191-0422220c3b31","Type":"ContainerDied","Data":"30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7"}
Sep 10 13:05:44.473710 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.473646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt" event={"ID":"9192d4a4-cf79-412d-b191-0422220c3b31","Type":"ContainerDied","Data":"aacf5c0e22918e0c83ba10dfe8029e1438716c0c12e69a0373466e388552ab0c"}
Sep 10 13:05:44.473710 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.473675    2577 scope.go:117] "RemoveContainer" containerID="30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7"
Sep 10 13:05:44.475050 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.475020    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e7075cf-07bf-45b8-9038-de3a36389db0" containerID="cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde" exitCode=0
Sep 10 13:05:44.475163 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.475072    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"
Sep 10 13:05:44.475163 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.475109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq" event={"ID":"3e7075cf-07bf-45b8-9038-de3a36389db0","Type":"ContainerDied","Data":"cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde"}
Sep 10 13:05:44.475163 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.475159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq" event={"ID":"3e7075cf-07bf-45b8-9038-de3a36389db0","Type":"ContainerDied","Data":"0fa155735ff08f571efd57bd2688a14c45baf221e3f10edb43e629966084a849"}
Sep 10 13:05:44.483538 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.483517    2577 scope.go:117] "RemoveContainer" containerID="30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7"
Sep 10 13:05:44.483901 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:05:44.483870    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7\": container with ID starting with 30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7 not found: ID does not exist" containerID="30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7"
Sep 10 13:05:44.483965 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.483905    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7"} err="failed to get container status \"30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7\": rpc error: code = NotFound desc = could not find container \"30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7\": container with ID starting with 30d018fe3f6c9370d2fb26d85af6c690c36332a9dbb40b524ba40988e9ae11c7 not found: ID does not exist"
Sep 10 13:05:44.483965 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.483928    2577 scope.go:117] "RemoveContainer" containerID="cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde"
Sep 10 13:05:44.494838 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.494818    2577 scope.go:117] "RemoveContainer" containerID="cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde"
Sep 10 13:05:44.495199 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:05:44.495167    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde\": container with ID starting with cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde not found: ID does not exist" containerID="cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde"
Sep 10 13:05:44.495322 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.495202    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde"} err="failed to get container status \"cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde\": rpc error: code = NotFound desc = could not find container \"cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde\": container with ID starting with cccfef44f0e9e68fee64036132e979dc9c975fff46a05045ebc6a54dbcddcdde not found: ID does not exist"
Sep 10 13:05:44.505096 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.505070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7075cf-07bf-45b8-9038-de3a36389db0-plugins-config-volume\") pod \"3e7075cf-07bf-45b8-9038-de3a36389db0\" (UID: \"3e7075cf-07bf-45b8-9038-de3a36389db0\") "
Sep 10 13:05:44.505201 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.505114    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9192d4a4-cf79-412d-b191-0422220c3b31-plugins-config-volume\") pod \"9192d4a4-cf79-412d-b191-0422220c3b31\" (UID: \"9192d4a4-cf79-412d-b191-0422220c3b31\") "
Sep 10 13:05:44.505265 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.505205    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzcpm\" (UniqueName: \"kubernetes.io/projected/9192d4a4-cf79-412d-b191-0422220c3b31-kube-api-access-qzcpm\") pod \"9192d4a4-cf79-412d-b191-0422220c3b31\" (UID: \"9192d4a4-cf79-412d-b191-0422220c3b31\") "
Sep 10 13:05:44.505323 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.505311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz2fv\" (UniqueName: \"kubernetes.io/projected/3e7075cf-07bf-45b8-9038-de3a36389db0-kube-api-access-tz2fv\") pod \"3e7075cf-07bf-45b8-9038-de3a36389db0\" (UID: \"3e7075cf-07bf-45b8-9038-de3a36389db0\") "
Sep 10 13:05:44.505474 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.505444    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7075cf-07bf-45b8-9038-de3a36389db0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e7075cf-07bf-45b8-9038-de3a36389db0" (UID: "3e7075cf-07bf-45b8-9038-de3a36389db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:05:44.505474 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.505450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9192d4a4-cf79-412d-b191-0422220c3b31-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9192d4a4-cf79-412d-b191-0422220c3b31" (UID: "9192d4a4-cf79-412d-b191-0422220c3b31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:05:44.505697 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.505675    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7075cf-07bf-45b8-9038-de3a36389db0-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:05:44.505697 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.505697    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9192d4a4-cf79-412d-b191-0422220c3b31-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:05:44.507288 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.507266    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9192d4a4-cf79-412d-b191-0422220c3b31-kube-api-access-qzcpm" (OuterVolumeSpecName: "kube-api-access-qzcpm") pod "9192d4a4-cf79-412d-b191-0422220c3b31" (UID: "9192d4a4-cf79-412d-b191-0422220c3b31"). InnerVolumeSpecName "kube-api-access-qzcpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:05:44.507375 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.507315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7075cf-07bf-45b8-9038-de3a36389db0-kube-api-access-tz2fv" (OuterVolumeSpecName: "kube-api-access-tz2fv") pod "3e7075cf-07bf-45b8-9038-de3a36389db0" (UID: "3e7075cf-07bf-45b8-9038-de3a36389db0"). InnerVolumeSpecName "kube-api-access-tz2fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:05:44.578158 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.578091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:44.607086 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.607059    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcpm\" (UniqueName: \"kubernetes.io/projected/9192d4a4-cf79-412d-b191-0422220c3b31-kube-api-access-qzcpm\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:05:44.607086 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.607081    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz2fv\" (UniqueName: \"kubernetes.io/projected/3e7075cf-07bf-45b8-9038-de3a36389db0-kube-api-access-tz2fv\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:05:44.677803 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.677759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:44.680051 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.680019    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"]
Sep 10 13:05:44.682090 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.682071    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c8b83b-rjfnm"]
Sep 10 13:05:44.808922 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.808893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"]
Sep 10 13:05:44.816360 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.816335    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c8b83b-7b44dcf4c8-rj7qt"]
Sep 10 13:05:44.831068 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.831038    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"]
Sep 10 13:05:44.834644 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:44.834616    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c8b83b-8c7c87647-42kfq"]
Sep 10 13:05:45.048265 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:45.048178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:46.001287 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:46.001253    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7075cf-07bf-45b8-9038-de3a36389db0" path="/var/lib/kubelet/pods/3e7075cf-07bf-45b8-9038-de3a36389db0/volumes"
Sep 10 13:05:46.001648 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:46.001633    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9192d4a4-cf79-412d-b191-0422220c3b31" path="/var/lib/kubelet/pods/9192d4a4-cf79-412d-b191-0422220c3b31/volumes"
Sep 10 13:05:46.001933 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:46.001921    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f37bf498-7788-4d04-9c0b-f78c2de6162a" path="/var/lib/kubelet/pods/f37bf498-7788-4d04-9c0b-f78c2de6162a/volumes"
Sep 10 13:05:52.853896 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:52.853855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:52.990323 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:52.990287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:53.301146 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:53.301118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:55.132062 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:55.132032    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:57.039909 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.039872    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"]
Sep 10 13:05:57.040345 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040300    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9192d4a4-cf79-412d-b191-0422220c3b31" containerName="payload-pre-processing"
Sep 10 13:05:57.040345 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040314    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9192d4a4-cf79-412d-b191-0422220c3b31" containerName="payload-pre-processing"
Sep 10 13:05:57.040345 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040337    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7075cf-07bf-45b8-9038-de3a36389db0" containerName="payload-processing"
Sep 10 13:05:57.040345 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040343    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7075cf-07bf-45b8-9038-de3a36389db0" containerName="payload-processing"
Sep 10 13:05:57.040480 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040361    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f37bf498-7788-4d04-9c0b-f78c2de6162a" containerName="revoke-keys"
Sep 10 13:05:57.040480 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040367    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f37bf498-7788-4d04-9c0b-f78c2de6162a" containerName="revoke-keys"
Sep 10 13:05:57.040480 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040425    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9192d4a4-cf79-412d-b191-0422220c3b31" containerName="payload-pre-processing"
Sep 10 13:05:57.040480 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040433    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f37bf498-7788-4d04-9c0b-f78c2de6162a" containerName="revoke-keys"
Sep 10 13:05:57.040480 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.040442    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7075cf-07bf-45b8-9038-de3a36389db0" containerName="payload-processing"
Sep 10 13:05:57.045217 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.045199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"
Sep 10 13:05:57.048837 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.048820    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 13:05:57.055656 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.055633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"]
Sep 10 13:05:57.200775 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.200743    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:57.219040 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.219010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fa56c3-ee6c-43db-a1e5-a8d6484639c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp\" (UID: \"29fa56c3-ee6c-43db-a1e5-a8d6484639c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"
Sep 10 13:05:57.320222 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.320143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fa56c3-ee6c-43db-a1e5-a8d6484639c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp\" (UID: \"29fa56c3-ee6c-43db-a1e5-a8d6484639c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"
Sep 10 13:05:57.320751 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.320732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fa56c3-ee6c-43db-a1e5-a8d6484639c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp\" (UID: \"29fa56c3-ee6c-43db-a1e5-a8d6484639c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"
Sep 10 13:05:57.356028 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.356004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"
Sep 10 13:05:57.636694 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:57.636659    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"]
Sep 10 13:05:58.550785 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:58.550753    2577 generic.go:356] "Generic (PLEG): container finished" podID="29fa56c3-ee6c-43db-a1e5-a8d6484639c6" containerID="f72fcb5f7a3d16c800378237bb78bb7e8c2dd35a2bd745e0da1352d8dc68e58a" exitCode=0
Sep 10 13:05:58.551143 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:58.550802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp" event={"ID":"29fa56c3-ee6c-43db-a1e5-a8d6484639c6","Type":"ContainerDied","Data":"f72fcb5f7a3d16c800378237bb78bb7e8c2dd35a2bd745e0da1352d8dc68e58a"}
Sep 10 13:05:58.551143 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:58.550846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp" event={"ID":"29fa56c3-ee6c-43db-a1e5-a8d6484639c6","Type":"ContainerStarted","Data":"5ed32dae3f29fa5dd4107c50287d6481737401a5f07b7f8be50394d90ccfec47"}
Sep 10 13:05:59.657269 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:59.657240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:05:59.692263 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:59.692241    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"
Sep 10 13:05:59.845858 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:59.845778    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fa56c3-ee6c-43db-a1e5-a8d6484639c6-maas-api-service-ca\") pod \"29fa56c3-ee6c-43db-a1e5-a8d6484639c6\" (UID: \"29fa56c3-ee6c-43db-a1e5-a8d6484639c6\") "
Sep 10 13:05:59.846172 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:59.846153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fa56c3-ee6c-43db-a1e5-a8d6484639c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29fa56c3-ee6c-43db-a1e5-a8d6484639c6" (UID: "29fa56c3-ee6c-43db-a1e5-a8d6484639c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:05:59.946725 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:05:59.946687    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fa56c3-ee6c-43db-a1e5-a8d6484639c6-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:06:00.564452 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:00.564413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp" event={"ID":"29fa56c3-ee6c-43db-a1e5-a8d6484639c6","Type":"ContainerDied","Data":"5ed32dae3f29fa5dd4107c50287d6481737401a5f07b7f8be50394d90ccfec47"}
Sep 10 13:06:00.564452 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:00.564449    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"
Sep 10 13:06:00.564686 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:00.564475    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ed32dae3f29fa5dd4107c50287d6481737401a5f07b7f8be50394d90ccfec47"
Sep 10 13:06:12.825266 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:12.825225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:12.857424 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:12.857390    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"]
Sep 10 13:06:12.857673 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:12.857645    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95" podUID="f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc" containerName="payload-pre-processing" containerID="cri-o://75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8" gracePeriod=30
Sep 10 13:06:12.860558 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:12.860514    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"]
Sep 10 13:06:12.860794 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:12.860767    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz" podUID="aee60f3a-a8a1-4338-924c-5a8765de78f5" containerName="payload-processing" containerID="cri-o://2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3" gracePeriod=30
Sep 10 13:06:12.915844 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:12.915815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:13.011240 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.011195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:13.100259 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.100206    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"]
Sep 10 13:06:13.103409 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.103386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:13.104525 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.104506    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c91a45-4kwcp"]
Sep 10 13:06:13.117752 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.117735    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:06:13.141235 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.141213    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:06:13.176232 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.176204    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-plugins-config-volume\") pod \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\" (UID: \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\") "
Sep 10 13:06:13.176346 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.176270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j628j\" (UniqueName: \"kubernetes.io/projected/aee60f3a-a8a1-4338-924c-5a8765de78f5-kube-api-access-j628j\") pod \"aee60f3a-a8a1-4338-924c-5a8765de78f5\" (UID: \"aee60f3a-a8a1-4338-924c-5a8765de78f5\") "
Sep 10 13:06:13.176346 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.176335    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee60f3a-a8a1-4338-924c-5a8765de78f5-plugins-config-volume\") pod \"aee60f3a-a8a1-4338-924c-5a8765de78f5\" (UID: \"aee60f3a-a8a1-4338-924c-5a8765de78f5\") "
Sep 10 13:06:13.176441 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.176405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4cmg\" (UniqueName: \"kubernetes.io/projected/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-kube-api-access-j4cmg\") pod \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\" (UID: \"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc\") "
Sep 10 13:06:13.176716 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.176565    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc" (UID: "f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:06:13.176837 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.176754    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee60f3a-a8a1-4338-924c-5a8765de78f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aee60f3a-a8a1-4338-924c-5a8765de78f5" (UID: "aee60f3a-a8a1-4338-924c-5a8765de78f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:06:13.178479 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.178450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-kube-api-access-j4cmg" (OuterVolumeSpecName: "kube-api-access-j4cmg") pod "f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc" (UID: "f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc"). InnerVolumeSpecName "kube-api-access-j4cmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:06:13.178869 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.178846    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee60f3a-a8a1-4338-924c-5a8765de78f5-kube-api-access-j628j" (OuterVolumeSpecName: "kube-api-access-j628j") pod "aee60f3a-a8a1-4338-924c-5a8765de78f5" (UID: "aee60f3a-a8a1-4338-924c-5a8765de78f5"). InnerVolumeSpecName "kube-api-access-j628j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:06:13.183773 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.183748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:13.277104 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.277076    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee60f3a-a8a1-4338-924c-5a8765de78f5-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:06:13.277104 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.277103    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4cmg\" (UniqueName: \"kubernetes.io/projected/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-kube-api-access-j4cmg\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:06:13.277252 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.277116    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:06:13.277252 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.277128    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j628j\" (UniqueName: \"kubernetes.io/projected/aee60f3a-a8a1-4338-924c-5a8765de78f5-kube-api-access-j628j\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:06:13.522455 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.522427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:13.632269 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.632237    2577 generic.go:356] "Generic (PLEG): container finished" podID="aee60f3a-a8a1-4338-924c-5a8765de78f5" containerID="2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3" exitCode=0
Sep 10 13:06:13.632458 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.632277    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"
Sep 10 13:06:13.632458 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.632314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz" event={"ID":"aee60f3a-a8a1-4338-924c-5a8765de78f5","Type":"ContainerDied","Data":"2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3"}
Sep 10 13:06:13.632458 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.632379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz" event={"ID":"aee60f3a-a8a1-4338-924c-5a8765de78f5","Type":"ContainerDied","Data":"76f120b9ff7edec843f473326cf1c708ed05d69ca0445a6bb46767006c3b771f"}
Sep 10 13:06:13.632458 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.632409    2577 scope.go:117] "RemoveContainer" containerID="2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3"
Sep 10 13:06:13.633853 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.633828    2577 generic.go:356] "Generic (PLEG): container finished" podID="f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc" containerID="75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8" exitCode=0
Sep 10 13:06:13.633966 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.633859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95" event={"ID":"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc","Type":"ContainerDied","Data":"75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8"}
Sep 10 13:06:13.633966 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.633873    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"
Sep 10 13:06:13.633966 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.633889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95" event={"ID":"f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc","Type":"ContainerDied","Data":"2aa2ff4fba4eb7b27061a58779de81429932b5b0dc465dc70b94cf6b040cb03d"}
Sep 10 13:06:13.641584 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.641568    2577 scope.go:117] "RemoveContainer" containerID="2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3"
Sep 10 13:06:13.641951 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:06:13.641929    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3\": container with ID starting with 2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3 not found: ID does not exist" containerID="2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3"
Sep 10 13:06:13.642060 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.641957    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3"} err="failed to get container status \"2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3\": rpc error: code = NotFound desc = could not find container \"2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3\": container with ID starting with 2f9309a65658a32599f690f57ec05f95680397c5696481921caa9da69e3cdda3 not found: ID does not exist"
Sep 10 13:06:13.642060 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.641978    2577 scope.go:117] "RemoveContainer" containerID="75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8"
Sep 10 13:06:13.650005 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.649988    2577 scope.go:117] "RemoveContainer" containerID="75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8"
Sep 10 13:06:13.650268 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:06:13.650245    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8\": container with ID starting with 75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8 not found: ID does not exist" containerID="75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8"
Sep 10 13:06:13.650360 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.650272    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8"} err="failed to get container status \"75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8\": rpc error: code = NotFound desc = could not find container \"75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8\": container with ID starting with 75a4fddf5715f9637ed06b52265f4a20a7c3fc4ef961c4ebc7f3831674a02bd8 not found: ID does not exist"
Sep 10 13:06:13.662133 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.662111    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"]
Sep 10 13:06:13.667398 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.667378    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c91a45-5d64798cc4-2qw95"]
Sep 10 13:06:13.681039 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.681018    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"]
Sep 10 13:06:13.686271 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:13.686249    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c91a45-b9979db67-zzzqz"]
Sep 10 13:06:14.000318 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:14.000278    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29fa56c3-ee6c-43db-a1e5-a8d6484639c6" path="/var/lib/kubelet/pods/29fa56c3-ee6c-43db-a1e5-a8d6484639c6/volumes"
Sep 10 13:06:14.000817 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:14.000798    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee60f3a-a8a1-4338-924c-5a8765de78f5" path="/var/lib/kubelet/pods/aee60f3a-a8a1-4338-924c-5a8765de78f5/volumes"
Sep 10 13:06:14.001182 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:14.001162    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc" path="/var/lib/kubelet/pods/f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc/volumes"
Sep 10 13:06:18.622799 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:18.622726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:39.271581 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:39.271543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:55.673673 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:55.673634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:06:57.792976 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:06:57.792926    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:04.043563 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:04.043523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:08.118946 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:08.118909    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:16.862370 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:16.862318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:16.960850 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:16.960809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:17.288419 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.288373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:17.649721 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.649637    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"]
Sep 10 13:07:17.650083 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650068    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee60f3a-a8a1-4338-924c-5a8765de78f5" containerName="payload-processing"
Sep 10 13:07:17.650133 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650085    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee60f3a-a8a1-4338-924c-5a8765de78f5" containerName="payload-processing"
Sep 10 13:07:17.650133 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650099    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc" containerName="payload-pre-processing"
Sep 10 13:07:17.650133 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650104    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc" containerName="payload-pre-processing"
Sep 10 13:07:17.650133 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650116    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29fa56c3-ee6c-43db-a1e5-a8d6484639c6" containerName="revoke-keys"
Sep 10 13:07:17.650133 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650121    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29fa56c3-ee6c-43db-a1e5-a8d6484639c6" containerName="revoke-keys"
Sep 10 13:07:17.650296 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650186    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3858d0a-fb6b-4021-8fb0-0a2eb8a60efc" containerName="payload-pre-processing"
Sep 10 13:07:17.650296 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650194    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29fa56c3-ee6c-43db-a1e5-a8d6484639c6" containerName="revoke-keys"
Sep 10 13:07:17.650296 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.650202    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee60f3a-a8a1-4338-924c-5a8765de78f5" containerName="payload-processing"
Sep 10 13:07:17.653770 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.653754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"
Sep 10 13:07:17.656973 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.656950    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 13:07:17.673032 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.673008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"]
Sep 10 13:07:17.765588 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.765555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b19f7ba-0389-48d2-be66-44caf0fc8d5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9\" (UID: \"7b19f7ba-0389-48d2-be66-44caf0fc8d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"
Sep 10 13:07:17.842202 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.842168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:17.866566 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.866540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b19f7ba-0389-48d2-be66-44caf0fc8d5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9\" (UID: \"7b19f7ba-0389-48d2-be66-44caf0fc8d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"
Sep 10 13:07:17.867160 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.867140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b19f7ba-0389-48d2-be66-44caf0fc8d5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9\" (UID: \"7b19f7ba-0389-48d2-be66-44caf0fc8d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"
Sep 10 13:07:17.964114 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:17.964043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"
Sep 10 13:07:18.237120 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:18.237096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"]
Sep 10 13:07:18.239448 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:07:18.239422    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b19f7ba_0389_48d2_be66_44caf0fc8d5a.slice/crio-cdbe9e58bd1ad13be96e026943100e74004faf8dc42fd51332b68155271d357f WatchSource:0}: Error finding container cdbe9e58bd1ad13be96e026943100e74004faf8dc42fd51332b68155271d357f: Status 404 returned error can't find the container with id cdbe9e58bd1ad13be96e026943100e74004faf8dc42fd51332b68155271d357f
Sep 10 13:07:18.978881 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:18.978850    2577 generic.go:356] "Generic (PLEG): container finished" podID="7b19f7ba-0389-48d2-be66-44caf0fc8d5a" containerID="e7ae088fcf02c48e8c57e66711f4e5f8214dbea1fdc1d06daca8cec8f5ed51ed" exitCode=0
Sep 10 13:07:18.979238 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:18.978927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9" event={"ID":"7b19f7ba-0389-48d2-be66-44caf0fc8d5a","Type":"ContainerDied","Data":"e7ae088fcf02c48e8c57e66711f4e5f8214dbea1fdc1d06daca8cec8f5ed51ed"}
Sep 10 13:07:18.979238 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:18.978977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9" event={"ID":"7b19f7ba-0389-48d2-be66-44caf0fc8d5a","Type":"ContainerStarted","Data":"cdbe9e58bd1ad13be96e026943100e74004faf8dc42fd51332b68155271d357f"}
Sep 10 13:07:20.117157 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:20.117134    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"
Sep 10 13:07:20.286838 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:20.286812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b19f7ba-0389-48d2-be66-44caf0fc8d5a-maas-api-service-ca\") pod \"7b19f7ba-0389-48d2-be66-44caf0fc8d5a\" (UID: \"7b19f7ba-0389-48d2-be66-44caf0fc8d5a\") "
Sep 10 13:07:20.287167 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:20.287141    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b19f7ba-0389-48d2-be66-44caf0fc8d5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b19f7ba-0389-48d2-be66-44caf0fc8d5a" (UID: "7b19f7ba-0389-48d2-be66-44caf0fc8d5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:07:20.388328 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:20.388297    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b19f7ba-0389-48d2-be66-44caf0fc8d5a-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:20.991418 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:20.991385    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"
Sep 10 13:07:20.991622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:20.991427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9" event={"ID":"7b19f7ba-0389-48d2-be66-44caf0fc8d5a","Type":"ContainerDied","Data":"cdbe9e58bd1ad13be96e026943100e74004faf8dc42fd51332b68155271d357f"}
Sep 10 13:07:20.991622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:20.991483    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdbe9e58bd1ad13be96e026943100e74004faf8dc42fd51332b68155271d357f"
Sep 10 13:07:28.392097 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.392061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:28.441095 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.441056    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"]
Sep 10 13:07:28.441303 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.441267    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n" podUID="693f8d1c-ebb8-4b8d-ab38-08e72a634b7c" containerName="payload-processing" containerID="cri-o://f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8" gracePeriod=30
Sep 10 13:07:28.458680 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.458542    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"]
Sep 10 13:07:28.458821 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.458794    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8" podUID="78d5215d-ce6f-4593-9c89-511d23946685" containerName="payload-pre-processing" containerID="cri-o://bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a" gracePeriod=30
Sep 10 13:07:28.489737 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.489699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:28.593661 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.593629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:28.697293 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.697259    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:28.699447 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.699418    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"]
Sep 10 13:07:28.705057 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.705033    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6ea780-7w6j9"]
Sep 10 13:07:28.705530 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.705513    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:07:28.730645 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.730622    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:07:28.760845 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.760822    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5215d-ce6f-4593-9c89-511d23946685-plugins-config-volume\") pod \"78d5215d-ce6f-4593-9c89-511d23946685\" (UID: \"78d5215d-ce6f-4593-9c89-511d23946685\") "
Sep 10 13:07:28.760983 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.760889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-plugins-config-volume\") pod \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\" (UID: \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\") "
Sep 10 13:07:28.761046 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.760999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vccwz\" (UniqueName: \"kubernetes.io/projected/78d5215d-ce6f-4593-9c89-511d23946685-kube-api-access-vccwz\") pod \"78d5215d-ce6f-4593-9c89-511d23946685\" (UID: \"78d5215d-ce6f-4593-9c89-511d23946685\") "
Sep 10 13:07:28.761108 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.761061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qwrw\" (UniqueName: \"kubernetes.io/projected/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-kube-api-access-8qwrw\") pod \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\" (UID: \"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c\") "
Sep 10 13:07:28.761516 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.761229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d5215d-ce6f-4593-9c89-511d23946685-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78d5215d-ce6f-4593-9c89-511d23946685" (UID: "78d5215d-ce6f-4593-9c89-511d23946685"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:07:28.761516 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.761229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "693f8d1c-ebb8-4b8d-ab38-08e72a634b7c" (UID: "693f8d1c-ebb8-4b8d-ab38-08e72a634b7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:07:28.761516 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.761501    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5215d-ce6f-4593-9c89-511d23946685-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:28.761755 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.761521    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:28.763947 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.763919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-kube-api-access-8qwrw" (OuterVolumeSpecName: "kube-api-access-8qwrw") pod "693f8d1c-ebb8-4b8d-ab38-08e72a634b7c" (UID: "693f8d1c-ebb8-4b8d-ab38-08e72a634b7c"). InnerVolumeSpecName "kube-api-access-8qwrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:07:28.764032 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.763951    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d5215d-ce6f-4593-9c89-511d23946685-kube-api-access-vccwz" (OuterVolumeSpecName: "kube-api-access-vccwz") pod "78d5215d-ce6f-4593-9c89-511d23946685" (UID: "78d5215d-ce6f-4593-9c89-511d23946685"). InnerVolumeSpecName "kube-api-access-vccwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:07:28.770912 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.770875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:28.863033 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.862998    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vccwz\" (UniqueName: \"kubernetes.io/projected/78d5215d-ce6f-4593-9c89-511d23946685-kube-api-access-vccwz\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:28.863033 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:28.863027    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qwrw\" (UniqueName: \"kubernetes.io/projected/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c-kube-api-access-8qwrw\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:29.032477 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.032444    2577 generic.go:356] "Generic (PLEG): container finished" podID="693f8d1c-ebb8-4b8d-ab38-08e72a634b7c" containerID="f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8" exitCode=0
Sep 10 13:07:29.032669 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.032489    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"
Sep 10 13:07:29.032669 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.032527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n" event={"ID":"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c","Type":"ContainerDied","Data":"f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8"}
Sep 10 13:07:29.032669 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.032587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n" event={"ID":"693f8d1c-ebb8-4b8d-ab38-08e72a634b7c","Type":"ContainerDied","Data":"dc2f08cd3e7419bac512061b1714f6138342531ad6486ffef68d311fc7a80031"}
Sep 10 13:07:29.032669 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.032638    2577 scope.go:117] "RemoveContainer" containerID="f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8"
Sep 10 13:07:29.034197 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.034174    2577 generic.go:356] "Generic (PLEG): container finished" podID="78d5215d-ce6f-4593-9c89-511d23946685" containerID="bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a" exitCode=0
Sep 10 13:07:29.034294 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.034218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8" event={"ID":"78d5215d-ce6f-4593-9c89-511d23946685","Type":"ContainerDied","Data":"bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a"}
Sep 10 13:07:29.034294 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.034258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8" event={"ID":"78d5215d-ce6f-4593-9c89-511d23946685","Type":"ContainerDied","Data":"3beb3b0da70b5116fb0a4903c920c4f2008606d95c5f96ddc49b9266681844c1"}
Sep 10 13:07:29.034294 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.034223    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"
Sep 10 13:07:29.045492 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.045055    2577 scope.go:117] "RemoveContainer" containerID="f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8"
Sep 10 13:07:29.045492 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:07:29.045453    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8\": container with ID starting with f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8 not found: ID does not exist" containerID="f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8"
Sep 10 13:07:29.045656 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.045503    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8"} err="failed to get container status \"f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8\": rpc error: code = NotFound desc = could not find container \"f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8\": container with ID starting with f106b7faa3b5c63a43896272e73f11fa5174074ddf93aca450df58ee1dc94dc8 not found: ID does not exist"
Sep 10 13:07:29.045656 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.045527    2577 scope.go:117] "RemoveContainer" containerID="bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a"
Sep 10 13:07:29.058313 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.058288    2577 scope.go:117] "RemoveContainer" containerID="bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a"
Sep 10 13:07:29.058754 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:07:29.058726    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a\": container with ID starting with bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a not found: ID does not exist" containerID="bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a"
Sep 10 13:07:29.058835 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.058761    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a"} err="failed to get container status \"bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a\": rpc error: code = NotFound desc = could not find container \"bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a\": container with ID starting with bbc94c0af48df61662e58fccf014071cc7d40a6c783d0eb017192b674995ff6a not found: ID does not exist"
Sep 10 13:07:29.068718 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.068692    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"]
Sep 10 13:07:29.073912 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.073888    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ea780-6d65fdf65-qc57n"]
Sep 10 13:07:29.087600 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.087580    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"]
Sep 10 13:07:29.093318 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.093298    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ea780-86c4d6445d-4ghs8"]
Sep 10 13:07:29.103261 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:29.103242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:30.000356 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:30.000319    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="693f8d1c-ebb8-4b8d-ab38-08e72a634b7c" path="/var/lib/kubelet/pods/693f8d1c-ebb8-4b8d-ab38-08e72a634b7c/volumes"
Sep 10 13:07:30.000744 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:30.000710    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78d5215d-ce6f-4593-9c89-511d23946685" path="/var/lib/kubelet/pods/78d5215d-ce6f-4593-9c89-511d23946685/volumes"
Sep 10 13:07:30.001006 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:30.000993    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b19f7ba-0389-48d2-be66-44caf0fc8d5a" path="/var/lib/kubelet/pods/7b19f7ba-0389-48d2-be66-44caf0fc8d5a/volumes"
Sep 10 13:07:35.913409 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:35.913358    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:36.011336 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.011306    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:36.219092 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.219008    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:36.594872 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.594841    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"]
Sep 10 13:07:36.595243 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595231    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693f8d1c-ebb8-4b8d-ab38-08e72a634b7c" containerName="payload-processing"
Sep 10 13:07:36.595289 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595245    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="693f8d1c-ebb8-4b8d-ab38-08e72a634b7c" containerName="payload-processing"
Sep 10 13:07:36.595289 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595269    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d5215d-ce6f-4593-9c89-511d23946685" containerName="payload-pre-processing"
Sep 10 13:07:36.595289 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595275    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d5215d-ce6f-4593-9c89-511d23946685" containerName="payload-pre-processing"
Sep 10 13:07:36.595289 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595285    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b19f7ba-0389-48d2-be66-44caf0fc8d5a" containerName="revoke-keys"
Sep 10 13:07:36.595419 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595291    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b19f7ba-0389-48d2-be66-44caf0fc8d5a" containerName="revoke-keys"
Sep 10 13:07:36.595419 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595347    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="693f8d1c-ebb8-4b8d-ab38-08e72a634b7c" containerName="payload-processing"
Sep 10 13:07:36.595419 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595355    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d5215d-ce6f-4593-9c89-511d23946685" containerName="payload-pre-processing"
Sep 10 13:07:36.595419 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.595366    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b19f7ba-0389-48d2-be66-44caf0fc8d5a" containerName="revoke-keys"
Sep 10 13:07:36.598534 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.598516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"
Sep 10 13:07:36.601481 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.601461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 13:07:36.608542 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.608513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"]
Sep 10 13:07:36.632679 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.632650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051ddf-4775-4281-9040-e64061bfb6dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc\" (UID: \"6a051ddf-4775-4281-9040-e64061bfb6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"
Sep 10 13:07:36.717149 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.717115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:36.733434 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.733408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051ddf-4775-4281-9040-e64061bfb6dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc\" (UID: \"6a051ddf-4775-4281-9040-e64061bfb6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"
Sep 10 13:07:36.734134 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.734111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051ddf-4775-4281-9040-e64061bfb6dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc\" (UID: \"6a051ddf-4775-4281-9040-e64061bfb6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"
Sep 10 13:07:36.909432 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:36.909349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"
Sep 10 13:07:37.197350 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:37.197305    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"]
Sep 10 13:07:37.197877 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:07:37.197843    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a051ddf_4775_4281_9040_e64061bfb6dd.slice/crio-b4498714b115bb3af3fb1b4db9a1910aff0dce8d0bd3fe77116187e5d0b10592 WatchSource:0}: Error finding container b4498714b115bb3af3fb1b4db9a1910aff0dce8d0bd3fe77116187e5d0b10592: Status 404 returned error can't find the container with id b4498714b115bb3af3fb1b4db9a1910aff0dce8d0bd3fe77116187e5d0b10592
Sep 10 13:07:38.086521 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:38.086487    2577 generic.go:356] "Generic (PLEG): container finished" podID="6a051ddf-4775-4281-9040-e64061bfb6dd" containerID="9e29bc112562df07684e93e315d729d0b284a44ee74427d99adbe8d8d73eaee9" exitCode=0
Sep 10 13:07:38.086724 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:38.086565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc" event={"ID":"6a051ddf-4775-4281-9040-e64061bfb6dd","Type":"ContainerDied","Data":"9e29bc112562df07684e93e315d729d0b284a44ee74427d99adbe8d8d73eaee9"}
Sep 10 13:07:38.086724 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:38.086644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc" event={"ID":"6a051ddf-4775-4281-9040-e64061bfb6dd","Type":"ContainerStarted","Data":"b4498714b115bb3af3fb1b4db9a1910aff0dce8d0bd3fe77116187e5d0b10592"}
Sep 10 13:07:39.221570 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:39.221541    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"
Sep 10 13:07:39.260553 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:39.260523    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051ddf-4775-4281-9040-e64061bfb6dd-maas-api-service-ca\") pod \"6a051ddf-4775-4281-9040-e64061bfb6dd\" (UID: \"6a051ddf-4775-4281-9040-e64061bfb6dd\") "
Sep 10 13:07:39.260931 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:39.260905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a051ddf-4775-4281-9040-e64061bfb6dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a051ddf-4775-4281-9040-e64061bfb6dd" (UID: "6a051ddf-4775-4281-9040-e64061bfb6dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:07:39.362295 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:39.362207    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051ddf-4775-4281-9040-e64061bfb6dd-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:40.098277 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:40.098247    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"
Sep 10 13:07:40.098472 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:40.098247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc" event={"ID":"6a051ddf-4775-4281-9040-e64061bfb6dd","Type":"ContainerDied","Data":"b4498714b115bb3af3fb1b4db9a1910aff0dce8d0bd3fe77116187e5d0b10592"}
Sep 10 13:07:40.098472 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:40.098410    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4498714b115bb3af3fb1b4db9a1910aff0dce8d0bd3fe77116187e5d0b10592"
Sep 10 13:07:47.516655 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.516618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:47.542496 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.542468    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"]
Sep 10 13:07:47.550171 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.550146    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c5adae-29817420-z7lc6"]
Sep 10 13:07:47.610359 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.610326    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"]
Sep 10 13:07:47.610700 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.610623    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh" podUID="c6d84e5f-6754-4ff2-ae30-535be48da7d9" containerName="payload-processing" containerID="cri-o://5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9" gracePeriod=30
Sep 10 13:07:47.625888 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.625865    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"]
Sep 10 13:07:47.626070 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.626051    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td" podUID="03291b8e-838f-44d4-a9b8-909fd648737e" containerName="payload-pre-processing" containerID="cri-o://b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d" gracePeriod=30
Sep 10 13:07:47.707820 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.707758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:47.789582 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.789541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:47.870257 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.870232    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 13:07:47.900837 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.900812    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 13:07:47.940375 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.940338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:47.942563 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.942534    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk7xd\" (UniqueName: \"kubernetes.io/projected/c6d84e5f-6754-4ff2-ae30-535be48da7d9-kube-api-access-vk7xd\") pod \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\" (UID: \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\") "
Sep 10 13:07:47.942730 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.942644    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d84e5f-6754-4ff2-ae30-535be48da7d9-plugins-config-volume\") pod \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\" (UID: \"c6d84e5f-6754-4ff2-ae30-535be48da7d9\") "
Sep 10 13:07:47.943032 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.943009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d84e5f-6754-4ff2-ae30-535be48da7d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6d84e5f-6754-4ff2-ae30-535be48da7d9" (UID: "c6d84e5f-6754-4ff2-ae30-535be48da7d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:07:47.944744 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.944723    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d84e5f-6754-4ff2-ae30-535be48da7d9-kube-api-access-vk7xd" (OuterVolumeSpecName: "kube-api-access-vk7xd") pod "c6d84e5f-6754-4ff2-ae30-535be48da7d9" (UID: "c6d84e5f-6754-4ff2-ae30-535be48da7d9"). InnerVolumeSpecName "kube-api-access-vk7xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:07:47.975948 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.975916    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"]
Sep 10 13:07:47.980455 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:47.980429    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c5adae-89pxc"]
Sep 10 13:07:48.000591 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.000519    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="651aa5a6-0a33-471d-98fc-c3c40a994cd6" path="/var/lib/kubelet/pods/651aa5a6-0a33-471d-98fc-c3c40a994cd6/volumes"
Sep 10 13:07:48.000892 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.000877    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a051ddf-4775-4281-9040-e64061bfb6dd" path="/var/lib/kubelet/pods/6a051ddf-4775-4281-9040-e64061bfb6dd/volumes"
Sep 10 13:07:48.043853 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.043776    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c967\" (UniqueName: \"kubernetes.io/projected/03291b8e-838f-44d4-a9b8-909fd648737e-kube-api-access-4c967\") pod \"03291b8e-838f-44d4-a9b8-909fd648737e\" (UID: \"03291b8e-838f-44d4-a9b8-909fd648737e\") "
Sep 10 13:07:48.043853 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.043824    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03291b8e-838f-44d4-a9b8-909fd648737e-plugins-config-volume\") pod \"03291b8e-838f-44d4-a9b8-909fd648737e\" (UID: \"03291b8e-838f-44d4-a9b8-909fd648737e\") "
Sep 10 13:07:48.044039 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.044027    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7xd\" (UniqueName: \"kubernetes.io/projected/c6d84e5f-6754-4ff2-ae30-535be48da7d9-kube-api-access-vk7xd\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:48.044091 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.044040    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d84e5f-6754-4ff2-ae30-535be48da7d9-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:48.044257 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.044225    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03291b8e-838f-44d4-a9b8-909fd648737e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03291b8e-838f-44d4-a9b8-909fd648737e" (UID: "03291b8e-838f-44d4-a9b8-909fd648737e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:07:48.045850 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.045830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03291b8e-838f-44d4-a9b8-909fd648737e-kube-api-access-4c967" (OuterVolumeSpecName: "kube-api-access-4c967") pod "03291b8e-838f-44d4-a9b8-909fd648737e" (UID: "03291b8e-838f-44d4-a9b8-909fd648737e"). InnerVolumeSpecName "kube-api-access-4c967". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:07:48.101362 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.101331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:48.140540 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.140512    2577 generic.go:356] "Generic (PLEG): container finished" podID="c6d84e5f-6754-4ff2-ae30-535be48da7d9" containerID="5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9" exitCode=0
Sep 10 13:07:48.140709 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.140555    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"
Sep 10 13:07:48.140709 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.140605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh" event={"ID":"c6d84e5f-6754-4ff2-ae30-535be48da7d9","Type":"ContainerDied","Data":"5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9"}
Sep 10 13:07:48.140709 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.140660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh" event={"ID":"c6d84e5f-6754-4ff2-ae30-535be48da7d9","Type":"ContainerDied","Data":"75a30b43e30f3344b960df8a46d9647225d583c553400b2dedd99a8cb520599e"}
Sep 10 13:07:48.140709 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.140685    2577 scope.go:117] "RemoveContainer" containerID="5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9"
Sep 10 13:07:48.142222 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.142202    2577 generic.go:356] "Generic (PLEG): container finished" podID="03291b8e-838f-44d4-a9b8-909fd648737e" containerID="b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d" exitCode=0
Sep 10 13:07:48.142331 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.142241    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"
Sep 10 13:07:48.142331 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.142253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td" event={"ID":"03291b8e-838f-44d4-a9b8-909fd648737e","Type":"ContainerDied","Data":"b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d"}
Sep 10 13:07:48.142331 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.142295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td" event={"ID":"03291b8e-838f-44d4-a9b8-909fd648737e","Type":"ContainerDied","Data":"aefd2b088929d5e22fb9b53cb8924c50e79886c6b8824e93e77eae2d0ce06e3e"}
Sep 10 13:07:48.144790 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.144755    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03291b8e-838f-44d4-a9b8-909fd648737e-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:48.144790 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.144785    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c967\" (UniqueName: \"kubernetes.io/projected/03291b8e-838f-44d4-a9b8-909fd648737e-kube-api-access-4c967\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:07:48.150634 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.150610    2577 scope.go:117] "RemoveContainer" containerID="5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9"
Sep 10 13:07:48.150967 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:07:48.150945    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9\": container with ID starting with 5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9 not found: ID does not exist" containerID="5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9"
Sep 10 13:07:48.151052 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.150972    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9"} err="failed to get container status \"5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9\": rpc error: code = NotFound desc = could not find container \"5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9\": container with ID starting with 5819659a29b0d650708bb91cc08cc06ef9a5140a65237adc06e87be3c5fd95e9 not found: ID does not exist"
Sep 10 13:07:48.151052 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.150991    2577 scope.go:117] "RemoveContainer" containerID="b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d"
Sep 10 13:07:48.158979 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.158963    2577 scope.go:117] "RemoveContainer" containerID="b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d"
Sep 10 13:07:48.159227 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:07:48.159207    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d\": container with ID starting with b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d not found: ID does not exist" containerID="b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d"
Sep 10 13:07:48.159285 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.159236    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d"} err="failed to get container status \"b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d\": rpc error: code = NotFound desc = could not find container \"b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d\": container with ID starting with b3c3333785adac37228ed88a9dc2ddca0d039fc36fee393e4ee1314c8b04575d not found: ID does not exist"
Sep 10 13:07:48.184135 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.184112    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"]
Sep 10 13:07:48.189373 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.189352    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c5adae-6785f544f-7frqh"]
Sep 10 13:07:48.210740 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.210713    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"]
Sep 10 13:07:48.215331 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.215310    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c5adae-5c7889674-h92td"]
Sep 10 13:07:48.395394 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:48.395325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:49.999914 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:49.999879    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03291b8e-838f-44d4-a9b8-909fd648737e" path="/var/lib/kubelet/pods/03291b8e-838f-44d4-a9b8-909fd648737e/volumes"
Sep 10 13:07:50.000297 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:50.000246    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d84e5f-6754-4ff2-ae30-535be48da7d9" path="/var/lib/kubelet/pods/c6d84e5f-6754-4ff2-ae30-535be48da7d9/volumes"
Sep 10 13:07:51.195752 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:51.195723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:54.270811 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:54.270773    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:56.090898 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.090851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:56.162466 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.162429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:56.505039 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.504951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:56.972975 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.972940    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"]
Sep 10 13:07:56.973460 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973440    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03291b8e-838f-44d4-a9b8-909fd648737e" containerName="payload-pre-processing"
Sep 10 13:07:56.973460 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973460    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="03291b8e-838f-44d4-a9b8-909fd648737e" containerName="payload-pre-processing"
Sep 10 13:07:56.973622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973471    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d84e5f-6754-4ff2-ae30-535be48da7d9" containerName="payload-processing"
Sep 10 13:07:56.973622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973477    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d84e5f-6754-4ff2-ae30-535be48da7d9" containerName="payload-processing"
Sep 10 13:07:56.973622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973485    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a051ddf-4775-4281-9040-e64061bfb6dd" containerName="revoke-keys"
Sep 10 13:07:56.973622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973490    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a051ddf-4775-4281-9040-e64061bfb6dd" containerName="revoke-keys"
Sep 10 13:07:56.973622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973550    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a051ddf-4775-4281-9040-e64061bfb6dd" containerName="revoke-keys"
Sep 10 13:07:56.973622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973559    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d84e5f-6754-4ff2-ae30-535be48da7d9" containerName="payload-processing"
Sep 10 13:07:56.973622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.973566    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="03291b8e-838f-44d4-a9b8-909fd648737e" containerName="payload-pre-processing"
Sep 10 13:07:56.978984 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.978968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"
Sep 10 13:07:56.982868 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:56.982848    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 13:07:57.010589 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:57.010568    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"]
Sep 10 13:07:57.123262 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:57.123230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e6a35a-898c-4ea4-ba44-099fcc24b111-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj\" (UID: \"98e6a35a-898c-4ea4-ba44-099fcc24b111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"
Sep 10 13:07:57.223844 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:57.223756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e6a35a-898c-4ea4-ba44-099fcc24b111-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj\" (UID: \"98e6a35a-898c-4ea4-ba44-099fcc24b111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"
Sep 10 13:07:57.224349 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:57.224328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e6a35a-898c-4ea4-ba44-099fcc24b111-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj\" (UID: \"98e6a35a-898c-4ea4-ba44-099fcc24b111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"
Sep 10 13:07:57.289522 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:57.289493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"
Sep 10 13:07:57.324347 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:57.324310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:07:57.595701 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:07:57.595666    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e6a35a_898c_4ea4_ba44_099fcc24b111.slice/crio-28a7ce4a6c32e09f2194a396c54d01e8eeaf4e844c62bd04ff91762bdb960327 WatchSource:0}: Error finding container 28a7ce4a6c32e09f2194a396c54d01e8eeaf4e844c62bd04ff91762bdb960327: Status 404 returned error can't find the container with id 28a7ce4a6c32e09f2194a396c54d01e8eeaf4e844c62bd04ff91762bdb960327
Sep 10 13:07:57.609236 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:57.609211    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"]
Sep 10 13:07:58.197971 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:58.197931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj" event={"ID":"98e6a35a-898c-4ea4-ba44-099fcc24b111","Type":"ContainerStarted","Data":"ee4a76e3a56e46ff5a12108953aa94de3f3ca5df57c17a3926dc7b4d35cd3e51"}
Sep 10 13:07:58.198361 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:58.197993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj" event={"ID":"98e6a35a-898c-4ea4-ba44-099fcc24b111","Type":"ContainerStarted","Data":"28a7ce4a6c32e09f2194a396c54d01e8eeaf4e844c62bd04ff91762bdb960327"}
Sep 10 13:07:58.229476 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:58.229428    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj" podStartSLOduration=2.229413845 podStartE2EDuration="2.229413845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:07:58.226268548 +0000 UTC m=+2252.748437600" watchObservedRunningTime="2026-09-10 13:07:58.229413845 +0000 UTC m=+2252.751582896"
Sep 10 13:07:59.204227 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:59.204192    2577 generic.go:356] "Generic (PLEG): container finished" podID="98e6a35a-898c-4ea4-ba44-099fcc24b111" containerID="ee4a76e3a56e46ff5a12108953aa94de3f3ca5df57c17a3926dc7b4d35cd3e51" exitCode=0
Sep 10 13:07:59.204692 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:07:59.204273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj" event={"ID":"98e6a35a-898c-4ea4-ba44-099fcc24b111","Type":"ContainerDied","Data":"ee4a76e3a56e46ff5a12108953aa94de3f3ca5df57c17a3926dc7b4d35cd3e51"}
Sep 10 13:08:00.338976 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:00.338952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"
Sep 10 13:08:00.451831 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:00.451799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e6a35a-898c-4ea4-ba44-099fcc24b111-maas-api-service-ca\") pod \"98e6a35a-898c-4ea4-ba44-099fcc24b111\" (UID: \"98e6a35a-898c-4ea4-ba44-099fcc24b111\") "
Sep 10 13:08:00.452173 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:00.452149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e6a35a-898c-4ea4-ba44-099fcc24b111-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98e6a35a-898c-4ea4-ba44-099fcc24b111" (UID: "98e6a35a-898c-4ea4-ba44-099fcc24b111"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:08:00.553134 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:00.553108    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e6a35a-898c-4ea4-ba44-099fcc24b111-maas-api-service-ca\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:08:01.215030 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:01.214995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj" event={"ID":"98e6a35a-898c-4ea4-ba44-099fcc24b111","Type":"ContainerDied","Data":"28a7ce4a6c32e09f2194a396c54d01e8eeaf4e844c62bd04ff91762bdb960327"}
Sep 10 13:08:01.215202 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:01.215050    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28a7ce4a6c32e09f2194a396c54d01e8eeaf4e844c62bd04ff91762bdb960327"
Sep 10 13:08:01.215202 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:01.215005    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"
Sep 10 13:08:07.478638 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.478573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:08:07.490976 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.490949    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"]
Sep 10 13:08:07.493830 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.493801    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e5208b-29817420-78kmx"]
Sep 10 13:08:07.541337 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.541302    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"]
Sep 10 13:08:07.541549 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.541512    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm" podUID="7f0dc5dd-69c5-430a-ba61-a53795548a74" containerName="payload-processing" containerID="cri-o://6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283" gracePeriod=30
Sep 10 13:08:07.543310 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.543284    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"]
Sep 10 13:08:07.543460 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.543440    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm" podUID="ccae0351-6797-4245-87a0-f04c89ab6aae" containerName="payload-pre-processing" containerID="cri-o://fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999" gracePeriod=30
Sep 10 13:08:07.631481 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.631450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:08:07.731543 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.731476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:08:07.772943 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.772875    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"]
Sep 10 13:08:07.775084 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.775013    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e5208b-nf8fj"]
Sep 10 13:08:07.798787 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.798764    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 13:08:07.825005 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.824967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:08:07.829168 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.829138    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 13:08:07.917350 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.917304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbqdv\" (UniqueName: \"kubernetes.io/projected/ccae0351-6797-4245-87a0-f04c89ab6aae-kube-api-access-cbqdv\") pod \"ccae0351-6797-4245-87a0-f04c89ab6aae\" (UID: \"ccae0351-6797-4245-87a0-f04c89ab6aae\") "
Sep 10 13:08:07.917350 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.917369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccae0351-6797-4245-87a0-f04c89ab6aae-plugins-config-volume\") pod \"ccae0351-6797-4245-87a0-f04c89ab6aae\" (UID: \"ccae0351-6797-4245-87a0-f04c89ab6aae\") "
Sep 10 13:08:07.917655 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.917414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4cwv\" (UniqueName: \"kubernetes.io/projected/7f0dc5dd-69c5-430a-ba61-a53795548a74-kube-api-access-p4cwv\") pod \"7f0dc5dd-69c5-430a-ba61-a53795548a74\" (UID: \"7f0dc5dd-69c5-430a-ba61-a53795548a74\") "
Sep 10 13:08:07.917655 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.917455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0dc5dd-69c5-430a-ba61-a53795548a74-plugins-config-volume\") pod \"7f0dc5dd-69c5-430a-ba61-a53795548a74\" (UID: \"7f0dc5dd-69c5-430a-ba61-a53795548a74\") "
Sep 10 13:08:07.917874 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.917838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccae0351-6797-4245-87a0-f04c89ab6aae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ccae0351-6797-4245-87a0-f04c89ab6aae" (UID: "ccae0351-6797-4245-87a0-f04c89ab6aae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:08:07.917874 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.917867    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0dc5dd-69c5-430a-ba61-a53795548a74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f0dc5dd-69c5-430a-ba61-a53795548a74" (UID: "7f0dc5dd-69c5-430a-ba61-a53795548a74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:08:07.919586 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.919567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccae0351-6797-4245-87a0-f04c89ab6aae-kube-api-access-cbqdv" (OuterVolumeSpecName: "kube-api-access-cbqdv") pod "ccae0351-6797-4245-87a0-f04c89ab6aae" (UID: "ccae0351-6797-4245-87a0-f04c89ab6aae"). InnerVolumeSpecName "kube-api-access-cbqdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:08:07.919660 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:07.919626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0dc5dd-69c5-430a-ba61-a53795548a74-kube-api-access-p4cwv" (OuterVolumeSpecName: "kube-api-access-p4cwv") pod "7f0dc5dd-69c5-430a-ba61-a53795548a74" (UID: "7f0dc5dd-69c5-430a-ba61-a53795548a74"). InnerVolumeSpecName "kube-api-access-p4cwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:08:08.000504 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.000433    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e6a35a-898c-4ea4-ba44-099fcc24b111" path="/var/lib/kubelet/pods/98e6a35a-898c-4ea4-ba44-099fcc24b111/volumes"
Sep 10 13:08:08.000795 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.000780    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb55da63-8b46-43fa-a7c1-6b5f034c6ef4" path="/var/lib/kubelet/pods/fb55da63-8b46-43fa-a7c1-6b5f034c6ef4/volumes"
Sep 10 13:08:08.018951 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.018929    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbqdv\" (UniqueName: \"kubernetes.io/projected/ccae0351-6797-4245-87a0-f04c89ab6aae-kube-api-access-cbqdv\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:08:08.019044 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.018953    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccae0351-6797-4245-87a0-f04c89ab6aae-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:08:08.019044 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.018968    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4cwv\" (UniqueName: \"kubernetes.io/projected/7f0dc5dd-69c5-430a-ba61-a53795548a74-kube-api-access-p4cwv\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:08:08.019044 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.018982    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0dc5dd-69c5-430a-ba61-a53795548a74-plugins-config-volume\") on node \"ip-10-0-140-169.ec2.internal\" DevicePath \"\""
Sep 10 13:08:08.160872 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.160841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:08:08.251633 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.251523    2577 generic.go:356] "Generic (PLEG): container finished" podID="7f0dc5dd-69c5-430a-ba61-a53795548a74" containerID="6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283" exitCode=0
Sep 10 13:08:08.251633 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.251566    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"
Sep 10 13:08:08.251873 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.251642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm" event={"ID":"7f0dc5dd-69c5-430a-ba61-a53795548a74","Type":"ContainerDied","Data":"6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283"}
Sep 10 13:08:08.251873 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.251702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm" event={"ID":"7f0dc5dd-69c5-430a-ba61-a53795548a74","Type":"ContainerDied","Data":"a221b27652d4188c41e53588d08028d34b6eba44b308541f5133fb3a74c9f73f"}
Sep 10 13:08:08.251873 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.251732    2577 scope.go:117] "RemoveContainer" containerID="6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283"
Sep 10 13:08:08.253268 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.253245    2577 generic.go:356] "Generic (PLEG): container finished" podID="ccae0351-6797-4245-87a0-f04c89ab6aae" containerID="fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999" exitCode=0
Sep 10 13:08:08.253352 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.253288    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"
Sep 10 13:08:08.253352 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.253312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm" event={"ID":"ccae0351-6797-4245-87a0-f04c89ab6aae","Type":"ContainerDied","Data":"fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999"}
Sep 10 13:08:08.253442 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.253354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm" event={"ID":"ccae0351-6797-4245-87a0-f04c89ab6aae","Type":"ContainerDied","Data":"06e2f24f939f231caca43b033ac939bc6f090a3023bc8f76ac0b0135c86a1755"}
Sep 10 13:08:08.260951 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.260935    2577 scope.go:117] "RemoveContainer" containerID="6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283"
Sep 10 13:08:08.261314 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:08:08.261289    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283\": container with ID starting with 6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283 not found: ID does not exist" containerID="6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283"
Sep 10 13:08:08.261364 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.261327    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283"} err="failed to get container status \"6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283\": rpc error: code = NotFound desc = could not find container \"6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283\": container with ID starting with 6b2c6e2a51769900984b3b0c6931a8f3dc26cf339a9658c9d7458d35b9b96283 not found: ID does not exist"
Sep 10 13:08:08.261364 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.261351    2577 scope.go:117] "RemoveContainer" containerID="fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999"
Sep 10 13:08:08.269037 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.269020    2577 scope.go:117] "RemoveContainer" containerID="fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999"
Sep 10 13:08:08.269289 ip-10-0-140-169 kubenswrapper[2577]: E0910 13:08:08.269271    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999\": container with ID starting with fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999 not found: ID does not exist" containerID="fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999"
Sep 10 13:08:08.269347 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.269297    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999"} err="failed to get container status \"fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999\": rpc error: code = NotFound desc = could not find container \"fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999\": container with ID starting with fffd01f3ad5bfc149a825065da6a38167d431584c4a0c7ee8213849084457999 not found: ID does not exist"
Sep 10 13:08:08.274041 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.274013    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"]
Sep 10 13:08:08.281211 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.281180    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e5208b-7bcc575989-lpdxm"]
Sep 10 13:08:08.295586 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.295557    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"]
Sep 10 13:08:08.300896 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:08.300879    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e5208b-5c4bd68946-xldnm"]
Sep 10 13:08:09.999799 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:09.999752    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f0dc5dd-69c5-430a-ba61-a53795548a74" path="/var/lib/kubelet/pods/7f0dc5dd-69c5-430a-ba61-a53795548a74/volumes"
Sep 10 13:08:10.000404 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:10.000375    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccae0351-6797-4245-87a0-f04c89ab6aae" path="/var/lib/kubelet/pods/ccae0351-6797-4245-87a0-f04c89ab6aae/volumes"
Sep 10 13:08:14.920946 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:14.920909    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:08:15.026173 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:15.026142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:08:15.225621 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:15.225526    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:08:26.497924 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:26.497893    2577 scope.go:117] "RemoveContainer" containerID="e0b1c77d0830b6afa8d1efcbbd1a114abc9f959d5559a1499583b3a7afa47f44"
Sep 10 13:08:26.506624 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:08:26.506578    2577 scope.go:117] "RemoveContainer" containerID="c6aa70f901913d5c670461aeca7cbf0c43ff5c984611effda17e6e180f49e47d"
Sep 10 13:09:42.392660 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:09:42.392622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:09:43.879832 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:09:43.879790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:09:45.436113 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:09:45.436077    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:10:26.128164 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:10:26.128138    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:10:26.133088 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:10:26.133065    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 13:10:26.135870 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:10:26.135852    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:10:26.140260 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:10:26.140243    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 13:11:15.817185 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:11:15.817135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:11:17.802984 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:11:17.802947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:12:26.574220 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:12:26.574190    2577 scope.go:117] "RemoveContainer" containerID="f72fcb5f7a3d16c800378237bb78bb7e8c2dd35a2bd745e0da1352d8dc68e58a"
Sep 10 13:12:26.582916 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:12:26.582899    2577 scope.go:117] "RemoveContainer" containerID="bb5246be5fb3166d04ffd52aa715d530f605b927a4355f8a006a5af3b46874bd"
Sep 10 13:12:42.712644 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:12:42.712578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:13:00.954730 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:13:00.954691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:13:26.608051 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:13:26.608016    2577 scope.go:117] "RemoveContainer" containerID="e7ae088fcf02c48e8c57e66711f4e5f8214dbea1fdc1d06daca8cec8f5ed51ed"
Sep 10 13:13:51.081773 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:13:51.081667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:13:52.078960 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:13:52.078925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:14:08.521032 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:08.521002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:14:26.627275 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:26.627242    2577 scope.go:117] "RemoveContainer" containerID="ee4a76e3a56e46ff5a12108953aa94de3f3ca5df57c17a3926dc7b4d35cd3e51"
Sep 10 13:14:26.635622 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:26.635588    2577 scope.go:117] "RemoveContainer" containerID="9e29bc112562df07684e93e315d729d0b284a44ee74427d99adbe8d8d73eaee9"
Sep 10 13:14:40.710749 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:40.710715    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77f75b9d49-6zcpp_6868b0f8-2576-4b45-9881-d5f1ff7c5c1c/authorino/0.log"
Sep 10 13:14:48.368368 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:48.368339    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-669b9dff44-fdj9c_e2717e90-2370-4dfe-9da9-7878f53e4df9/copy-manifests/0.log"
Sep 10 13:14:48.409826 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:48.409798    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-669b9dff44-fdj9c_e2717e90-2370-4dfe-9da9-7878f53e4df9/manager/0.log"
Sep 10 13:14:49.460129 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:49.460097    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qqc9h_07b56097-bd8f-4f0d-abc8-93833838c026/postgres/0.log"
Sep 10 13:14:50.323675 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:50.323642    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77f75b9d49-6zcpp_6868b0f8-2576-4b45-9881-d5f1ff7c5c1c/authorino/0.log"
Sep 10 13:14:51.842345 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:51.842316    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-669df89dc4-lrvxv_4dd94ead-5235-45ac-a072-49282503eb00/kube-auth-proxy/0.log"
Sep 10 13:14:52.104168 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:52.104089    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5cvsq_fb44708b-b4a0-42c0-8fff-592329cef1f5/payload-pre-processing/0.log"
Sep 10 13:14:52.235061 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:52.235032    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zc97k_68357e6d-f22c-40cf-b4b4-67bd383505e9/payload-processing/0.log"
Sep 10 13:14:52.362554 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:52.362482    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84884d55d8-8jzhd_029b95e7-f063-4791-9a5b-1f3db7ad0c3a/router/0.log"
Sep 10 13:14:52.850169 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:52.850140    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gpxpk_c8086638-f32f-4c2e-96a5-6394432f98b4/main/0.log"
Sep 10 13:14:53.212851 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:53.212767    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fj5fn9_826a241c-f0cf-4722-84be-292191eca9c2/main/0.log"
Sep 10 13:14:59.646506 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.646469    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fkh8h/must-gather-rmhrf"]
Sep 10 13:14:59.646918 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.646902    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0dc5dd-69c5-430a-ba61-a53795548a74" containerName="payload-processing"
Sep 10 13:14:59.646964 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.646920    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0dc5dd-69c5-430a-ba61-a53795548a74" containerName="payload-processing"
Sep 10 13:14:59.646964 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.646929    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e6a35a-898c-4ea4-ba44-099fcc24b111" containerName="revoke-keys"
Sep 10 13:14:59.646964 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.646935    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e6a35a-898c-4ea4-ba44-099fcc24b111" containerName="revoke-keys"
Sep 10 13:14:59.646964 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.646946    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccae0351-6797-4245-87a0-f04c89ab6aae" containerName="payload-pre-processing"
Sep 10 13:14:59.646964 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.646953    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccae0351-6797-4245-87a0-f04c89ab6aae" containerName="payload-pre-processing"
Sep 10 13:14:59.647123 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.647024    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccae0351-6797-4245-87a0-f04c89ab6aae" containerName="payload-pre-processing"
Sep 10 13:14:59.647123 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.647034    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e6a35a-898c-4ea4-ba44-099fcc24b111" containerName="revoke-keys"
Sep 10 13:14:59.647123 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.647042    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0dc5dd-69c5-430a-ba61-a53795548a74" containerName="payload-processing"
Sep 10 13:14:59.650070 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.650045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkh8h/must-gather-rmhrf"
Sep 10 13:14:59.650353 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.650322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:14:59.654513 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.654492    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fkh8h\"/\"default-dockercfg-dskqh\""
Sep 10 13:14:59.654816 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.654801    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fkh8h\"/\"kube-root-ca.crt\""
Sep 10 13:14:59.655620 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.655586    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fkh8h\"/\"openshift-service-ca.crt\""
Sep 10 13:14:59.671138 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.671120    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fkh8h/must-gather-rmhrf"]
Sep 10 13:14:59.769295 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.769266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1ba6e1b1-9770-466e-bb39-c15713a5da63-must-gather-output\") pod \"must-gather-rmhrf\" (UID: \"1ba6e1b1-9770-466e-bb39-c15713a5da63\") " pod="openshift-must-gather-fkh8h/must-gather-rmhrf"
Sep 10 13:14:59.769401 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.769320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp229\" (UniqueName: \"kubernetes.io/projected/1ba6e1b1-9770-466e-bb39-c15713a5da63-kube-api-access-dp229\") pod \"must-gather-rmhrf\" (UID: \"1ba6e1b1-9770-466e-bb39-c15713a5da63\") " pod="openshift-must-gather-fkh8h/must-gather-rmhrf"
Sep 10 13:14:59.840795 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.840775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log" limit=1024
Sep 10 13:14:59.870759 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.870738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1ba6e1b1-9770-466e-bb39-c15713a5da63-must-gather-output\") pod \"must-gather-rmhrf\" (UID: \"1ba6e1b1-9770-466e-bb39-c15713a5da63\") " pod="openshift-must-gather-fkh8h/must-gather-rmhrf"
Sep 10 13:14:59.870838 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.870779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp229\" (UniqueName: \"kubernetes.io/projected/1ba6e1b1-9770-466e-bb39-c15713a5da63-kube-api-access-dp229\") pod \"must-gather-rmhrf\" (UID: \"1ba6e1b1-9770-466e-bb39-c15713a5da63\") " pod="openshift-must-gather-fkh8h/must-gather-rmhrf"
Sep 10 13:14:59.871054 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.871034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1ba6e1b1-9770-466e-bb39-c15713a5da63-must-gather-output\") pod \"must-gather-rmhrf\" (UID: \"1ba6e1b1-9770-466e-bb39-c15713a5da63\") " pod="openshift-must-gather-fkh8h/must-gather-rmhrf"
Sep 10 13:14:59.884679 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.884655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp229\" (UniqueName: \"kubernetes.io/projected/1ba6e1b1-9770-466e-bb39-c15713a5da63-kube-api-access-dp229\") pod \"must-gather-rmhrf\" (UID: \"1ba6e1b1-9770-466e-bb39-c15713a5da63\") " pod="openshift-must-gather-fkh8h/must-gather-rmhrf"
Sep 10 13:14:59.959586 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:14:59.959509    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkh8h/must-gather-rmhrf"
Sep 10 13:15:00.250911 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:00.250821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fkh8h/must-gather-rmhrf"]
Sep 10 13:15:00.253489 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:00.253464    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c"]
Sep 10 13:15:00.255498 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:15:00.255476    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ba6e1b1_9770_466e_bb39_c15713a5da63.slice/crio-ad87b34ee78b574064730b13b3d0eea51cd6c56431f9731566567378986dfe4d WatchSource:0}: Error finding container ad87b34ee78b574064730b13b3d0eea51cd6c56431f9731566567378986dfe4d: Status 404 returned error can't find the container with id ad87b34ee78b574064730b13b3d0eea51cd6c56431f9731566567378986dfe4d
Sep 10 13:15:00.257769 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:00.257751    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:15:00.258485 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:00.258468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c"
Sep 10 13:15:00.261645 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:00.261624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 13:15:00.271260 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:00.271242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c"
Sep 10 13:15:00.277826 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:00.277796    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c"]
Sep 10 13:15:00.560788 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:00.560753    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c"]
Sep 10 13:15:00.563533 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:15:00.563501    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0713491_74e0_488d_81b0_a046b15f104d.slice/crio-a53b77ccf5955a2335b1f5f7183329638718986d8a78e81156f145dd349e86e2 WatchSource:0}: Error finding container a53b77ccf5955a2335b1f5f7183329638718986d8a78e81156f145dd349e86e2: Status 404 returned error can't find the container with id a53b77ccf5955a2335b1f5f7183329638718986d8a78e81156f145dd349e86e2
Sep 10 13:15:01.256680 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:01.256638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c" event={"ID":"d0713491-74e0-488d-81b0-a046b15f104d","Type":"ContainerStarted","Data":"ab748665a945abeb280ea32ca7bb25a103f909359a02dcd498f79bc9d7bac6a0"}
Sep 10 13:15:01.257117 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:01.256712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c" event={"ID":"d0713491-74e0-488d-81b0-a046b15f104d","Type":"ContainerStarted","Data":"a53b77ccf5955a2335b1f5f7183329638718986d8a78e81156f145dd349e86e2"}
Sep 10 13:15:01.258534 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:01.258504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkh8h/must-gather-rmhrf" event={"ID":"1ba6e1b1-9770-466e-bb39-c15713a5da63","Type":"ContainerStarted","Data":"9fc76bdb09a38df2f18857250db21c744c134c9e872582769f60c151179d8948"}
Sep 10 13:15:01.258676 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:01.258545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkh8h/must-gather-rmhrf" event={"ID":"1ba6e1b1-9770-466e-bb39-c15713a5da63","Type":"ContainerStarted","Data":"d0c7a168e1129f7e934962233fb2b0faa1e9bbe5839c83a25076308cf03035f0"}
Sep 10 13:15:01.258676 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:01.258565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkh8h/must-gather-rmhrf" event={"ID":"1ba6e1b1-9770-466e-bb39-c15713a5da63","Type":"ContainerStarted","Data":"ad87b34ee78b574064730b13b3d0eea51cd6c56431f9731566567378986dfe4d"}
Sep 10 13:15:01.280369 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:01.280316    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c" podStartSLOduration=1.28030125 podStartE2EDuration="1.28030125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:15:01.279649696 +0000 UTC m=+2675.801818748" watchObservedRunningTime="2026-09-10 13:15:01.28030125 +0000 UTC m=+2675.802470302"
Sep 10 13:15:01.315252 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:01.315191    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fkh8h/must-gather-rmhrf" podStartSLOduration=1.5746235450000001 podStartE2EDuration="2.315174305s" totalImagesPullingTime="740.55076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:15:01.311667415 +0000 UTC m=+2675.833836487" watchObservedRunningTime="2026-09-10 13:15:01.315174305 +0000 UTC m=+2675.837343357"
Sep 10 13:15:02.275638 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:02.275581    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0713491-74e0-488d-81b0-a046b15f104d" containerID="ab748665a945abeb280ea32ca7bb25a103f909359a02dcd498f79bc9d7bac6a0" exitCode=0
Sep 10 13:15:02.276006 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:02.275649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c" event={"ID":"d0713491-74e0-488d-81b0-a046b15f104d","Type":"ContainerDied","Data":"ab748665a945abeb280ea32ca7bb25a103f909359a02dcd498f79bc9d7bac6a0"}
Sep 10 13:15:03.144435 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:03.144387    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n4575_646110b4-56d3-44b3-b75c-3fb2883b274b/global-pull-secret-syncer/0.log"
Sep 10 13:15:03.298646 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:03.298577    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6zsm8_8f579811-e1ef-4422-b6cc-d78f644e771f/konnectivity-agent/0.log"
Sep 10 13:15:03.444033 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:03.444004    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c"
Sep 10 13:15:03.549018 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:03.548986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-169.ec2.internal_9ad92d93c55b1bd4bbad87ef6544190d/haproxy/0.log"
Sep 10 13:15:04.292493 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:04.292403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c" event={"ID":"d0713491-74e0-488d-81b0-a046b15f104d","Type":"ContainerDied","Data":"a53b77ccf5955a2335b1f5f7183329638718986d8a78e81156f145dd349e86e2"}
Sep 10 13:15:04.292493 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:04.292463    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53b77ccf5955a2335b1f5f7183329638718986d8a78e81156f145dd349e86e2"
Sep 10 13:15:04.292737 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:04.292529    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817435-hdf5c"
Sep 10 13:15:08.219670 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:08.219558    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77f75b9d49-6zcpp_6868b0f8-2576-4b45-9881-d5f1ff7c5c1c/authorino/0.log"
Sep 10 13:15:10.533533 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:10.533504    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-5k5lq_70f7e891-ffa9-4196-a488-e84d0ff56f7a/kube-state-metrics/0.log"
Sep 10 13:15:10.563968 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:10.563934    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-5k5lq_70f7e891-ffa9-4196-a488-e84d0ff56f7a/kube-rbac-proxy-main/0.log"
Sep 10 13:15:10.596539 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:10.596499    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-5k5lq_70f7e891-ffa9-4196-a488-e84d0ff56f7a/kube-rbac-proxy-self/0.log"
Sep 10 13:15:10.649457 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:10.649420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f74d6669d-4c55b_eacad1b9-2965-4fb4-8310-6d5a1e78a0fe/metrics-server/0.log"
Sep 10 13:15:10.951057 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:10.951028    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dk8b2_f1218ff9-94f8-493a-aef5-599b24fca4b9/node-exporter/0.log"
Sep 10 13:15:11.000987 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.000924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dk8b2_f1218ff9-94f8-493a-aef5-599b24fca4b9/kube-rbac-proxy/0.log"
Sep 10 13:15:11.044614 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.044568    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dk8b2_f1218ff9-94f8-493a-aef5-599b24fca4b9/init-textfile/0.log"
Sep 10 13:15:11.289117 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.289055    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-x5q99_06744632-29d8-4339-a789-1d53398c153a/kube-rbac-proxy-main/0.log"
Sep 10 13:15:11.328837 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.328803    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-x5q99_06744632-29d8-4339-a789-1d53398c153a/kube-rbac-proxy-self/0.log"
Sep 10 13:15:11.372438 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.372401    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-x5q99_06744632-29d8-4339-a789-1d53398c153a/openshift-state-metrics/0.log"
Sep 10 13:15:11.436844 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.436762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8/prometheus/0.log"
Sep 10 13:15:11.525282 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.525254    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8/config-reloader/0.log"
Sep 10 13:15:11.586106 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.586073    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8/thanos-sidecar/0.log"
Sep 10 13:15:11.732147 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.732070    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8/kube-rbac-proxy-web/0.log"
Sep 10 13:15:11.818079 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.818049    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8/kube-rbac-proxy/0.log"
Sep 10 13:15:11.921248 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:11.921209    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8/kube-rbac-proxy-thanos/0.log"
Sep 10 13:15:12.048726 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:12.048693    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_efbd6785-8bec-4ecc-9e6c-cfb008b8d7a8/init-config-reloader/0.log"
Sep 10 13:15:12.279770 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:12.279734    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-svb9h_64fb7a24-54f4-4086-b4fd-f502603a60c7/prometheus-operator-admission-webhook/0.log"
Sep 10 13:15:12.340123 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:12.340030    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b859c5658-rlzt4_256405e5-57f3-4649-9d2b-1389fc955e9a/telemeter-client/0.log"
Sep 10 13:15:12.401260 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:12.401230    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b859c5658-rlzt4_256405e5-57f3-4649-9d2b-1389fc955e9a/reload/0.log"
Sep 10 13:15:12.467944 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:12.467913    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b859c5658-rlzt4_256405e5-57f3-4649-9d2b-1389fc955e9a/kube-rbac-proxy/0.log"
Sep 10 13:15:13.468303 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.468256    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"]
Sep 10 13:15:13.468870 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.468842    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0713491-74e0-488d-81b0-a046b15f104d" containerName="cleanup"
Sep 10 13:15:13.468870 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.468867    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0713491-74e0-488d-81b0-a046b15f104d" containerName="cleanup"
Sep 10 13:15:13.469003 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.468976    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0713491-74e0-488d-81b0-a046b15f104d" containerName="cleanup"
Sep 10 13:15:13.476771 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.476746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.494046 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.494006    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"]
Sep 10 13:15:13.517671 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.517623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lvr\" (UniqueName: \"kubernetes.io/projected/c9aecb83-eec0-4a0e-987e-152350148f1e-kube-api-access-89lvr\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.517850 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.517683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-lib-modules\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.517850 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.517737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-sys\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.517850 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.517826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-proc\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.517995 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.517889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-podres\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.618960 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.618918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-sys\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.619148 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.619015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-proc\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.619148 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.619054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-sys\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.619148 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.619086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-podres\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.619148 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.619125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-proc\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.619353 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.619154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lvr\" (UniqueName: \"kubernetes.io/projected/c9aecb83-eec0-4a0e-987e-152350148f1e-kube-api-access-89lvr\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.619353 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.619181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-lib-modules\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.619353 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.619190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-podres\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.619353 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.619307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9aecb83-eec0-4a0e-987e-152350148f1e-lib-modules\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.630211 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.630183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lvr\" (UniqueName: \"kubernetes.io/projected/c9aecb83-eec0-4a0e-987e-152350148f1e-kube-api-access-89lvr\") pod \"perf-node-gather-daemonset-84lk7\" (UID: \"c9aecb83-eec0-4a0e-987e-152350148f1e\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:13.792233 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:13.792198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:14.095059 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:14.095028    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"]
Sep 10 13:15:14.098881 ip-10-0-140-169 kubenswrapper[2577]: W0910 13:15:14.098847    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc9aecb83_eec0_4a0e_987e_152350148f1e.slice/crio-4e1b0bc215f846c908cd74f787c54b3ff2452c56ca8e010e67301c708ef6f7ff WatchSource:0}: Error finding container 4e1b0bc215f846c908cd74f787c54b3ff2452c56ca8e010e67301c708ef6f7ff: Status 404 returned error can't find the container with id 4e1b0bc215f846c908cd74f787c54b3ff2452c56ca8e010e67301c708ef6f7ff
Sep 10 13:15:14.368057 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:14.367971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7" event={"ID":"c9aecb83-eec0-4a0e-987e-152350148f1e","Type":"ContainerStarted","Data":"01fd336a3b040afe17907531a8f561b5e8690203c6148aafc5a0eda1da44e8b0"}
Sep 10 13:15:14.368057 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:14.368034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7" event={"ID":"c9aecb83-eec0-4a0e-987e-152350148f1e","Type":"ContainerStarted","Data":"4e1b0bc215f846c908cd74f787c54b3ff2452c56ca8e010e67301c708ef6f7ff"}
Sep 10 13:15:14.368405 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:14.368204    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:14.739714 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:14.739636    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:15:14.748177 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:14.748142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/2.log"
Sep 10 13:15:15.357576 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:15.357542    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c666f94d8-d54f4_93c3c99d-52c4-4ae7-94a3-c5f7252acd7e/console/0.log"
Sep 10 13:15:15.428730 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:15.428692    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-tmlbj_fad3daa4-04a8-4bcb-a8a2-875b5262e073/download-server/0.log"
Sep 10 13:15:16.084217 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:16.084182    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-57crx_d0475af5-553d-4372-a2e1-0353f8f96578/volume-data-source-validator/0.log"
Sep 10 13:15:17.442431 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:17.442404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jf9x8_9dfc4909-2e6d-4b80-a781-ed7e8fa59616/dns/0.log"
Sep 10 13:15:17.489659 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:17.489627    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jf9x8_9dfc4909-2e6d-4b80-a781-ed7e8fa59616/kube-rbac-proxy/0.log"
Sep 10 13:15:17.590094 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:17.590067    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-59cz5_72c9ae18-ee7a-4949-b2ae-4de97e29e46a/dns-node-resolver/0.log"
Sep 10 13:15:18.398429 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:18.398351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rrqld_2a86ebb5-e828-4641-9ffd-c25ab4039f7e/node-ca/0.log"
Sep 10 13:15:19.972642 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:19.972606    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-669df89dc4-lrvxv_4dd94ead-5235-45ac-a072-49282503eb00/kube-auth-proxy/0.log"
Sep 10 13:15:20.089504 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:20.089452    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5cvsq_fb44708b-b4a0-42c0-8fff-592329cef1f5/payload-pre-processing/0.log"
Sep 10 13:15:20.137830 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:20.137795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zc97k_68357e6d-f22c-40cf-b4b4-67bd383505e9/payload-processing/0.log"
Sep 10 13:15:20.177285 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:20.177256    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84884d55d8-8jzhd_029b95e7-f063-4791-9a5b-1f3db7ad0c3a/router/0.log"
Sep 10 13:15:20.383255 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:20.383228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7"
Sep 10 13:15:20.429959 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:20.429912    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-84lk7" podStartSLOduration=7.429897106 podStartE2EDuration="7.429897106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:15:14.396563857 +0000 UTC m=+2688.918732921" watchObservedRunningTime="2026-09-10 13:15:20.429897106 +0000 UTC m=+2694.952066160"
Sep 10 13:15:21.833743 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:21.833710    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xthlv_ee744d16-fa03-4d53-98fe-cb6fea265c0c/serve-healthcheck-canary/0.log"
Sep 10 13:15:22.943713 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:22.943620    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2nh2p_52e6ec7a-6c26-4bea-a9da-7d29d4e036fc/kube-rbac-proxy/0.log"
Sep 10 13:15:23.023758 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:23.023681    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2nh2p_52e6ec7a-6c26-4bea-a9da-7d29d4e036fc/exporter/0.log"
Sep 10 13:15:23.086280 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:23.086252    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2nh2p_52e6ec7a-6c26-4bea-a9da-7d29d4e036fc/extractor/0.log"
Sep 10 13:15:23.774313 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:23.774277    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-669df89dc4-lrvxv_4dd94ead-5235-45ac-a072-49282503eb00/kube-auth-proxy/0.log"
Sep 10 13:15:23.865204 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:23.865172    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5cvsq_fb44708b-b4a0-42c0-8fff-592329cef1f5/payload-pre-processing/0.log"
Sep 10 13:15:23.977688 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:23.977588    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zc97k_68357e6d-f22c-40cf-b4b4-67bd383505e9/payload-processing/0.log"
Sep 10 13:15:24.038305 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:24.038264    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84884d55d8-8jzhd_029b95e7-f063-4791-9a5b-1f3db7ad0c3a/router/0.log"
Sep 10 13:15:26.101909 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:26.101880    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-669df89dc4-lrvxv_4dd94ead-5235-45ac-a072-49282503eb00/kube-auth-proxy/0.log"
Sep 10 13:15:26.173625 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:26.173082    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5cvsq_fb44708b-b4a0-42c0-8fff-592329cef1f5/payload-pre-processing/0.log"
Sep 10 13:15:26.177648 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:26.177628    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:15:26.184852 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:26.184827    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9kcpx_1a71518e-b89a-40af-bd74-3c444b7cabe6/console-operator/1.log"
Sep 10 13:15:26.185644 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:26.185624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 13:15:26.191085 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:26.191064    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftz99_fd93453e-1cad-4867-8586-95ed7eba199c/ovn-acl-logging/0.log"
Sep 10 13:15:26.217059 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:26.217022    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zc97k_68357e6d-f22c-40cf-b4b4-67bd383505e9/payload-processing/0.log"
Sep 10 13:15:26.238932 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:26.238909    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84884d55d8-8jzhd_029b95e7-f063-4791-9a5b-1f3db7ad0c3a/router/0.log"
Sep 10 13:15:27.416796 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:27.416761    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-669b9dff44-fdj9c_e2717e90-2370-4dfe-9da9-7878f53e4df9/manager/0.log"
Sep 10 13:15:27.449639 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:27.449610    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-669b9dff44-fdj9c_e2717e90-2370-4dfe-9da9-7878f53e4df9/copy-manifests/0.log"
Sep 10 13:15:28.781673 ip-10-0-140-169 kubenswrapper[2577]: I0910 13:15:28.781638    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d95565f4b-jxmgb_45c79074-852a-4c3e-9193-a9b359371584/manager/0.log"