Sep 09 21:36:39.464929 ip-10-0-139-221 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 21:36:39.464940 ip-10-0-139-221 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 21:36:39.464948 ip-10-0-139-221 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 21:36:39.465160 ip-10-0-139-221 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 21:36:49.695705 ip-10-0-139-221 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 21:36:49.695721 ip-10-0-139-221 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5dac19eb00fd469baa4322b7b06678ad --
Sep 09 21:38:44.234525 ip-10-0-139-221 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 21:38:44.640784 ip-10-0-139-221 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:38:44.640784 ip-10-0-139-221 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 21:38:44.640784 ip-10-0-139-221 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:38:44.640784 ip-10-0-139-221 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 21:38:44.640784 ip-10-0-139-221 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 21:38:44.641492 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.640851    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 21:38:44.648786 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648755    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:44.648786 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648779    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:44.648786 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648783    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:44.648786 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648786    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:44.648786 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648789    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:44.648786 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648793    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:44.648786 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648796    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:44.648786 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648799    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648803    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648806    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648809    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648812    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648815    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648817    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648820    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648822    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648825    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648828    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648831    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648833    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648836    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648839    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648841    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648844    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648846    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648851    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648855    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:44.649109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648857    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648860    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648863    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648866    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648868    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648871    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648874    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648876    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648879    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648881    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648884    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648888    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648892    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648896    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648900    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648909    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648912    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648915    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648918    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648920    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:44.649632 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648923    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648926    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648928    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648931    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648933    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648936    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648939    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648942    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648946    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648950    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648953    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648956    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648958    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648962    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648964    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648967    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648969    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648972    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648975    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:44.650150 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648977    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648980    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648982    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648985    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648987    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648990    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648992    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648996    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.648999    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649002    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649004    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649007    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649010    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649012    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649015    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649029    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649032    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649035    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649038    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649041    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:44.650629 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649044    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649046    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649050    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649053    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649056    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649059    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649062    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649065    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649067    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649070    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649073    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649075    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649078    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649803    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649810    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649814    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649817    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649820    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649823    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:44.651152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649827    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649829    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649832    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649835    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649837    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649840    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649843    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649846    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649849    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649851    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649853    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649857    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649859    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649862    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649865    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649869    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649872    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649876    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649879    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:44.651622 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649882    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649885    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649888    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649890    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649893    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649896    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649900    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649904    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649906    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649909    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649912    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649916    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649919    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649922    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649925    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649928    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649930    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649933    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649936    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649938    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:44.652160 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649941    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649943    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649946    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649948    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649951    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649955    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649957    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649960    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649962    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649965    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649967    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649970    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649972    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649975    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649977    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649980    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649982    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649985    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649987    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649990    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:44.652656 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649994    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.649997    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650000    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650003    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650006    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650009    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650012    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650014    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650031    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650034    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650036    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650039    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650042    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650044    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650048    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650052    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650055    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650058    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650062    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:44.653164 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650064    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650067    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650070    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650073    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650075    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650078    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650081    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650083    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650086    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650089    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650091    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650094    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650097    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650099    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650102    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650515    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650521    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650523    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650526    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650529    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:44.653636 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650532    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650535    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650538    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650540    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650544    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650546    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650549    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650552    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650554    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650557    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650560    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650566    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650569    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650572    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650575    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650577    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650580    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650583    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650585    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650588    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:44.654152 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650591    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650593    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650596    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650598    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650601    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650603    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650606    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650608    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650612    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650614    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650617    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650620    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650622    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650627    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650630    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650633    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650637    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650639    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650642    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:44.654657 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650645    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650648    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650650    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650653    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650656    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650659    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650662    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650664    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650667    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650669    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650672    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650674    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650677    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650679    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650682    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650684    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650687    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650689    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650692    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650694    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:44.655147 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650697    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650700    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650703    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650706    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650708    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650711    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650713    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650716    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650718    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650721    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650723    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650728    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650732    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650736    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650739    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650743    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650746    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650749    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650752    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650755    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:44.655644 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650758    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650761    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650764    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650767    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650769    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650772    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650774    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650777    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650779    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650782    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650785    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650787    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650790    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650795    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.650797    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650883    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650933    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650942    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650947    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650951    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650955    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 21:38:44.656161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650959    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650964    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650967    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650970    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650974    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650977    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650980    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650983    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650986    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.650989    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651000    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651004    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651008    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651010    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651014    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651029    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651033    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651042    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651045    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651049    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651052    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651055    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651058    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651061    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 21:38:44.656675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651067    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651070    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651075    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651078    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651081    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651084    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651087    2579 flags.go:64] FLAG: --enable-server="true"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651090    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651095    2579 flags.go:64] FLAG: --event-burst="100"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651098    2579 flags.go:64] FLAG: --event-qps="50"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651101    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651104    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651107    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651111    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651114    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651117    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651120    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651123    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651126    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651130    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651133    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651136    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651139    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651142    2579 flags.go:64] FLAG: --feature-gates=""
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651146    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 21:38:44.657258 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651149    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651152    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651156    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651159    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651162    2579 flags.go:64] FLAG: --help="false"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651165    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-139-221.ec2.internal"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651168    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651173    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651177    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651180    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651184    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651187    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651190    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651193    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651195    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651198    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651201    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651204    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651207    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651210    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651213    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651216    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651218    2579 flags.go:64] FLAG: --lock-file=""
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651221    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 21:38:44.657957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651224    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651227    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651233    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651237    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651240    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651242    2579 flags.go:64] FLAG: --logging-format="text"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651245    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651249    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651251    2579 flags.go:64] FLAG: --manifest-url=""
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651254    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651259    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651262    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651271    2579 flags.go:64] FLAG: --max-pods="110"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651274    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651277    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651282    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651285    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651288    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651291    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651294    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651303    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651306    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651309    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 21:38:44.658577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651312    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651315    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651322    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651325    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651328    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651331    2579 flags.go:64] FLAG: --port="10250"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651334    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651337    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05f747896c9263714"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651341    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651344    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651347    2579 flags.go:64] FLAG: --register-node="true"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651350    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651354    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651357    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651360    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651363    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651367    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651370    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651373    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651376    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651379    2579 flags.go:64] FLAG: --runonce="false"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651381    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651385    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651388    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651394    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 21:38:44.659175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651398    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651401    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651405    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651408    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651411    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651414    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651417    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651420    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651423    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651426    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651429    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651434    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651437    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651440    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651444    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651447    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651450    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651452    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651455    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651459    2579 flags.go:64] FLAG: --v="2"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651463    2579 flags.go:64] FLAG: --version="false"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651467    2579 flags.go:64] FLAG: --vmodule=""
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651471    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651475    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651597    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:44.659798 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651601    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651604    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651607    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651610    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651613    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651616    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651620    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651625    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651630    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651634    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651637    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651641    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651643    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651646    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651649    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651652    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651654    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651658    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651660    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651663    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:44.660429 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651666    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651669    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651671    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651674    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651677    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651680    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651682    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651685    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651687    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651691    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651694    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651696    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651699    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651701    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651704    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651707    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651710    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651712    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651716    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651720    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:44.660937 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651723    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651725    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651728    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651730    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651733    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651736    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651738    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651741    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651744    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651746    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651749    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651751    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651754    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651757    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651760    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651762    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651765    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651767    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651770    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:44.661530 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651774    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651778    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651781    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651784    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651787    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651791    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651794    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651796    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651799    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651802    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651805    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651809    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651812    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651815    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651819    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651821    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651824    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651828    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651831    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:44.662006 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651834    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651837    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651840    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651844    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651847    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651850    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651852    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651855    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651858    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651860    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651863    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651865    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651868    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651871    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651873    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651876    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651879    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651882    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651884    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:44.662592 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.651887    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.651895    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.659192    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.659208    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.660043    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.660055    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660124    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660130    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660133    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660137    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660140    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660143    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660146    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660150    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:44.663097 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660153    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660156    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660159    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660163    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660167    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660170    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660173    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660176    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660179    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660181    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660184    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660186    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660189    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660191    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660194    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660197    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660200    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660203    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660206    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660209    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:44.663463 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660211    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660215    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660218    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660221    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660223    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660226    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660229    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660231    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660234    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660237    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660239    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660242    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660244    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660247    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660250    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660252    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660256    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660258    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660261    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660264    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:44.663999 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660267    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660270    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660274    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660277    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660281    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660284    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660287    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660289    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660292    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660298    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660300    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660303    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660305    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660308    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660311    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660313    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660316    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660318    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660321    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:44.664601 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660323    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660326    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660329    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660331    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660334    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660336    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660339    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660342    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660344    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660347    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660349    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660352    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660354    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660357    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660359    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660362    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660365    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660368    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660371    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660373    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:44.665153 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660376    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660378    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660381    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660384    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660387    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660389    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660392    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660396    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660399    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660402    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660404    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660407    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.660412    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660523    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660527    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 21:38:44.665642 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660530    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660533    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660536    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660538    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660541    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660544    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660547    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660550    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660552    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660555    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660558    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660560    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660563    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660566    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660568    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660571    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660574    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660576    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660579    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660582    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 21:38:44.666013 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660584    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660587    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660590    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660592    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660595    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660597    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660600    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660603    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660605    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660608    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660611    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660613    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660616    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660619    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660621    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660624    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660626    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660629    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660631    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660634    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 21:38:44.666777 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660636    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660639    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660641    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660644    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660647    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660649    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660652    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660654    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660657    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660660    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660663    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660666    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660668    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660671    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660674    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660676    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660679    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660681    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660684    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 21:38:44.667299 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660686    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660688    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660691    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660693    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660696    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660700    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660703    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660706    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660708    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660711    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660714    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660717    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660719    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660722    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660725    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660728    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660730    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660733    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660738    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 21:38:44.667760 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660741    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660744    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660747    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660749    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660751    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660754    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660756    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660759    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660762    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660765    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660767    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660769    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660772    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660775    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660777    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660780    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660782    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660785    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 21:38:44.668245 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:44.660787    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 21:38:44.668702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.660793    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 21:38:44.668702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.660906    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 21:38:44.668702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.663800    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 21:38:44.668702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.664812    2579 server.go:1019] "Starting client certificate rotation"
Sep 09 21:38:44.668702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.664916    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 21:38:44.668702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.665726    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 21:38:44.668702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.668187    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 21:38:44.672007 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.671979    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 21:38:44.682967 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.682943    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 09 21:38:44.686116 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.686092    2579 log.go:25] "Validated CRI v1 image API"
Sep 09 21:38:44.686961 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.686943    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 21:38:44.689825 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.689797    2579 fs.go:135] Filesystem UUIDs: map[2b1abfbe-f538-430f-a772-3e5ac95f2b89:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 edfa9bd0-ea5a-4a3c-8185-6a519572088e:/dev/nvme0n1p4]
Sep 09 21:38:44.689870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.689825    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 21:38:44.694458 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.694437    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 21:38:44.695679 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.695563    2579 manager.go:216] Machine: {Timestamp:2026-09-09 21:38:44.694173692 +0000 UTC m=+0.340119266 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3136571 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c29442d7c18b94b78fdf25201b67 SystemUUID:ec21c294-42d7-c18b-94b7-8fdf25201b67 BootID:5dac19eb-00fd-469b-aa43-22b7b06678ad Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:eb:4b:58:ff:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:eb:4b:58:ff:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:26:aa:10:01:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 21:38:44.695679 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.695677    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 21:38:44.695814 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.695803    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 21:38:44.698741 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.698714    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 21:38:44.698906 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.698744    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 21:38:44.699921 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.699910    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 21:38:44.699961 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.699925    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 21:38:44.699961 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.699933    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 21:38:44.700647 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.700635    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 21:38:44.701880 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.701869    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 21:38:44.705641 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.705630    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 21:38:44.705761 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.705751    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 21:38:44.705795 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.705782    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 21:38:44.705795 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.705792    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 21:38:44.705852 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.705807    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 21:38:44.705852 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.705818    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 09 21:38:44.705852 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.705827    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 21:38:44.706322 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.706296    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 21:38:44.706322 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.706324    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 21:38:44.708694 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.708677    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 21:38:44.711132 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711117    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 21:38:44.711198 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711149    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 21:38:44.711316 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711306    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 21:38:44.711356 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711320    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 21:38:44.711356 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711326    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 21:38:44.711356 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711332    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 21:38:44.711356 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711340    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 21:38:44.711356 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711346    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 21:38:44.711921 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711912    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 21:38:44.711921 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711920    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 21:38:44.711990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711927    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 21:38:44.711990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711933    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 21:38:44.711990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711942    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 21:38:44.711990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.711951    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 21:38:44.712709 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.712697    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 21:38:44.712709 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.712706    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 21:38:44.716304 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.716283    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 21:38:44.716397 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.716384    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 21:38:44.716602 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.716579    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 21:38:44.716828 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.716815    2579 server.go:1268] "Started kubelet"
Sep 09 21:38:44.716910 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.716886    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 21:38:44.717441 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.717410    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 21:38:44.717532 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.717464    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 21:38:44.717890 ip-10-0-139-221 systemd[1]: Started Kubernetes Kubelet.
Sep 09 21:38:44.718079 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.718069    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 21:38:44.718663 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.718649    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 21:38:44.725262 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.724891    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 21:38:44.726823 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.726799    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 21:38:44.727330 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.727312    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 21:38:44.727430 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.727403    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:44.727495 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.726500    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-221.ec2.internal.18d3c4dba97f1fbb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-221.ec2.internal,UID:ip-10-0-139-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-221.ec2.internal,},FirstTimestamp:2026-09-09 21:38:44.716781499 +0000 UTC m=+0.362727073,LastTimestamp:2026-09-09 21:38:44.716781499 +0000 UTC m=+0.362727073,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-221.ec2.internal,}"
Sep 09 21:38:44.727971 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.727955    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 21:38:44.727971 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.727968    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 21:38:44.728100 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.727984    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 21:38:44.728100 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.728080    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 21:38:44.728100 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.728094    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 21:38:44.730301 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.730277    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8k9ll"
Sep 09 21:38:44.730384 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.730309    2579 factory.go:55] Registering systemd factory
Sep 09 21:38:44.730384 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.730335    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 09 21:38:44.730597 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.730586    2579 factory.go:153] Registering CRI-O factory
Sep 09 21:38:44.730630 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.730604    2579 factory.go:223] Registration of the crio container factory successfully
Sep 09 21:38:44.730671 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.730656    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 21:38:44.730704 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.730679    2579 factory.go:103] Registering Raw factory
Sep 09 21:38:44.730704 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.730691    2579 manager.go:1196] Started watching for new ooms in manager
Sep 09 21:38:44.731071 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.731060    2579 manager.go:349] Starting recovery of all containers
Sep 09 21:38:44.735257 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.735234    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8k9ll"
Sep 09 21:38:44.736066 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.736042    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 21:38:44.736066 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.736043    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 21:38:44.736172 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.736105    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 21:38:44.743054 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.743032    2579 manager.go:354] Recovery completed
Sep 09 21:38:44.749196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.749182    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:44.751248 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.751228    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:44.751340 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.751265    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:44.751340 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.751282    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:44.751873 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.751856    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 21:38:44.751873 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.751866    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:44.751990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.751872    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 21:38:44.751990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.751909    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 21:38:44.752602 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.752583    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:44.752671 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.752626    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:44.752671 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.752642    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:44.752955 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.752874    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-221.ec2.internal.18d3c4dbab8d0abb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-221.ec2.internal,UID:ip-10-0-139-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-221.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-221.ec2.internal,},FirstTimestamp:2026-09-09 21:38:44.751248059 +0000 UTC m=+0.397193638,LastTimestamp:2026-09-09 21:38:44.751248059 +0000 UTC m=+0.397193638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-221.ec2.internal,}"
Sep 09 21:38:44.753818 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.753806    2579 policy_none.go:49] "None policy: Start"
Sep 09 21:38:44.753871 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.753822    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 21:38:44.753871 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.753833    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 21:38:44.755653 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.755643    2579 policy_none.go:47] "Start"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.793597    2579 manager.go:337] "Starting Device Plugin manager"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.793765    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.793777    2579 server.go:87] "Starting device plugin registration server"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.793978    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.793989    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.794160    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.794255    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.794269    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.794716    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 21:38:44.809705 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.794757    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:44.861645 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.861607    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 21:38:44.861797 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.861675    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 21:38:44.861797 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.861705    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 21:38:44.861797 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.861755    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 21:38:44.864032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.864000    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:44.895005 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.894920    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:44.896176 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.896155    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:44.896298 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.896194    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:44.896298 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.896210    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:44.896298 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.896241    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-221.ec2.internal"
Sep 09 21:38:44.906570 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.906550    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-221.ec2.internal"
Sep 09 21:38:44.906650 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.906575    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-221.ec2.internal\": node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:44.926611 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.926587    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:44.962561 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.962529    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal"]
Sep 09 21:38:44.962725 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.962616    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:44.963681 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.963664    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:44.963794 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.963698    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:44.963794 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.963711    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:44.965855 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.965840    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:44.965961 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.965946    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:44.966047 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.965975    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:44.966662 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.966641    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:44.966711 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.966665    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:44.966711 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.966679    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:44.967235 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.967216    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:44.967307 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.967248    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:44.967307 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.967260    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:44.969453 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.969439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:44.969525 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.969465    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 21:38:44.970256 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.970239    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 21:38:44.970353 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.970264    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 21:38:44.970353 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:44.970277    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeHasSufficientPID"
Sep 09 21:38:44.997911 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:44.997884    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-221.ec2.internal\" not found" node="ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.002285 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:45.002268    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-221.ec2.internal\" not found" node="ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.027517 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:45.027494    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:45.030776 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.030761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8ae9f8a9e6c826f807116759d8def20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal\" (UID: \"c8ae9f8a9e6c826f807116759d8def20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.030841 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.030784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ae9f8a9e6c826f807116759d8def20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal\" (UID: \"c8ae9f8a9e6c826f807116759d8def20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.030841 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.030801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1378741cd49bb53a4187c0b23b12755-config\") pod \"kube-apiserver-proxy-ip-10-0-139-221.ec2.internal\" (UID: \"a1378741cd49bb53a4187c0b23b12755\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.127956 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:45.127917    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:45.131183 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.131165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8ae9f8a9e6c826f807116759d8def20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal\" (UID: \"c8ae9f8a9e6c826f807116759d8def20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.131254 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.131192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ae9f8a9e6c826f807116759d8def20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal\" (UID: \"c8ae9f8a9e6c826f807116759d8def20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.131254 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.131227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1378741cd49bb53a4187c0b23b12755-config\") pod \"kube-apiserver-proxy-ip-10-0-139-221.ec2.internal\" (UID: \"a1378741cd49bb53a4187c0b23b12755\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.131332 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.131287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ae9f8a9e6c826f807116759d8def20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal\" (UID: \"c8ae9f8a9e6c826f807116759d8def20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.131367 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.131338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8ae9f8a9e6c826f807116759d8def20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal\" (UID: \"c8ae9f8a9e6c826f807116759d8def20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.131406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.131370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1378741cd49bb53a4187c0b23b12755-config\") pod \"kube-apiserver-proxy-ip-10-0-139-221.ec2.internal\" (UID: \"a1378741cd49bb53a4187c0b23b12755\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.228661 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:45.228571    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:45.300964 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.300924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.304659 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:45.304634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:45.329278 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:45.329249    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:46.653367 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:46.653332    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:46.753884 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:46.753803    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:46.824256 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:46.824229    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:46.854415 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:46.854375    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:46.888688 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:46.888659    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 21:38:46.889341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:46.888798    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 21:38:46.889341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:46.888836    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 21:38:46.954946 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:46.954910    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:46.962205 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:46.962176    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 21:33:45 +0000 UTC" deadline="2028-05-27 04:16:50.284942314 +0000 UTC"
Sep 09 21:38:46.962205 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:46.962201    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15006h38m3.322743701s"
Sep 09 21:38:47.055785 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:47.055713    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:47.156382 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:47.156352    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:47.216415 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:47.216386    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ae9f8a9e6c826f807116759d8def20.slice/crio-e8a3ed4a4c4ac54d7261ba5ac87a10200f6f22766f9ab3f942124b5ed9a0dab6 WatchSource:0}: Error finding container e8a3ed4a4c4ac54d7261ba5ac87a10200f6f22766f9ab3f942124b5ed9a0dab6: Status 404 returned error can't find the container with id e8a3ed4a4c4ac54d7261ba5ac87a10200f6f22766f9ab3f942124b5ed9a0dab6
Sep 09 21:38:47.216799 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:47.216782    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1378741cd49bb53a4187c0b23b12755.slice/crio-e4ef62a0efdd599c8fc5a8afcf19d3ee3fe19dfb627303d74425496206475b9e WatchSource:0}: Error finding container e4ef62a0efdd599c8fc5a8afcf19d3ee3fe19dfb627303d74425496206475b9e: Status 404 returned error can't find the container with id e4ef62a0efdd599c8fc5a8afcf19d3ee3fe19dfb627303d74425496206475b9e
Sep 09 21:38:47.223920 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.223902    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:38:47.256757 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:47.256723    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:47.357275 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:47.357202    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-221.ec2.internal\" not found"
Sep 09 21:38:47.367244 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.367222    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:47.414096 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.414066    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:47.452053 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.452026    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:47.463063 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.463040    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 21:38:47.464101 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.464085    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal"
Sep 09 21:38:47.471185 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.471170    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 21:38:47.930891 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.930832    2579 apiserver.go:52] "Watching apiserver"
Sep 09 21:38:47.939412 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.939383    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 21:38:47.941883 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.941089    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-vvvk2","openshift-multus/network-metrics-daemon-wvv59","openshift-network-operator/iptables-alerter-fzqbr","kube-system/global-pull-secret-syncer-9kpfp","openshift-dns/node-resolver-vg5zh","openshift-image-registry/node-ca-d8dsb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal","openshift-multus/multus-additional-cni-plugins-7k8zp","openshift-multus/multus-lqrz5","openshift-network-diagnostics/network-check-target-zdcfg","openshift-ovn-kubernetes/ovnkube-node-jjtjx","kube-system/konnectivity-agent-fpkj8","kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"]
Sep 09 21:38:47.943944 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.943919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:47.944054 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:47.944018    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:38:47.946260 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.946238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.948674 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.948550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:47.949054 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.948791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 21:38:47.949054 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.948882    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 21:38:47.949054 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.948916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t2scw\""
Sep 09 21:38:47.949241 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.949104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 21:38:47.949241 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.949143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 21:38:47.949336 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.949310    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 21:38:47.949518 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.949499    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 21:38:47.950908 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.950879    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 21:38:47.951003 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.950907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:38:47.951003 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.950889    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h5c9m\""
Sep 09 21:38:47.951162 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.951147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:47.951241 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:47.951220    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:38:47.951294 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.951250    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 21:38:47.951407 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.951386    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 21:38:47.953135 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.953114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:47.955550 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.955532    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 21:38:47.955651 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.955569    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 21:38:47.955710 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.955532    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sllv9\""
Sep 09 21:38:47.956679 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.956660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:47.959208 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.958676    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 21:38:47.959208 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.958744    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 21:38:47.959208 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.958966    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 21:38:47.959208 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.959064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:47.959208 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.959131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pwz2d\""
Sep 09 21:38:47.961558 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.961358    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2qpx5\""
Sep 09 21:38:47.962226 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.961781    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 21:38:47.962226 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.961922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:47.962226 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.961936    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 21:38:47.962226 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.962165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 21:38:47.962501 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.962411    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 21:38:47.963972 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.963951    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 21:38:47.964248 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964231    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 21:38:47.964332 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f86d9c73-bc49-4713-96dd-4b96e1cf9156-hosts-file\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:47.964332 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-systemd\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964443 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964443 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-cni-bin\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964443 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99c997d6-cc19-4559-8591-58462280463f-dbus\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:47.964443 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:47.964443 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-slash\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-env-overrides\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75434aec-7a98-4360-b03d-867428a3e8ae-ovn-node-metrics-cert\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-ovn\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49e4c0b3-6c4b-42ec-b83a-0df16108208a-iptables-alerter-script\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f86d9c73-bc49-4713-96dd-4b96e1cf9156-tmp-dir\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964666    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hczq7\""
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964680    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-ovnkube-config\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.964758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-var-lib-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-cni-netd\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-ovnkube-script-lib\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkv2l\" (UniqueName: \"kubernetes.io/projected/49e4c0b3-6c4b-42ec-b83a-0df16108208a-kube-api-access-mkv2l\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-run-netns\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99c997d6-cc19-4559-8591-58462280463f-kubelet-config\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:47.964917    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4dg\" (UniqueName: \"kubernetes.io/projected/f86d9c73-bc49-4713-96dd-4b96e1cf9156-kube-api-access-tj4dg\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:47.965170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.964993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-systemd-units\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.965856 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.965033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-node-log\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.967522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.966195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qtk\" (UniqueName: \"kubernetes.io/projected/75434aec-7a98-4360-b03d-867428a3e8ae-kube-api-access-n4qtk\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.967522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.966233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-kubelet\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.967522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.966259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49e4c0b3-6c4b-42ec-b83a-0df16108208a-host-slash\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:47.967522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.966281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-etc-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.967522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.966299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.967522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.966313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-log-socket\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:47.967522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.966872    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 21:38:47.967522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.966980    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:47.969276 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.969256    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:38:47.969360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.969257    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 21:38:47.969439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.969258    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nvrhq\""
Sep 09 21:38:47.969683 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.969664    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:47.972022 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.972001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:38:47.972147 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.972131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 21:38:47.972387 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.972368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 21:38:47.972478 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.972407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jk262\""
Sep 09 21:38:47.975421 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.975399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 21:38:47.975520 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.975506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 21:38:47.975590 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.975507    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dsvkz\""
Sep 09 21:38:47.990290 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.990265    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7q8jt"
Sep 09 21:38:47.998462 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:47.998440    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7q8jt"
Sep 09 21:38:48.052942 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.052911    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 21:38:48.067175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1e38884-a5a6-4a4e-9734-4415e7b28338-tmp\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.067345 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f86d9c73-bc49-4713-96dd-4b96e1cf9156-hosts-file\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:48.067345 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d166fa40-a07a-4ca2-9909-37ca14b614b3-cni-binary-copy\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.067345 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-etc-selinux\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.067345 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hd8q\" (UniqueName: \"kubernetes.io/projected/8db25aa0-8917-4c9c-8d86-855d010e4c13-kube-api-access-5hd8q\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.067345 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f86d9c73-bc49-4713-96dd-4b96e1cf9156-hosts-file\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:48.067585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-systemd\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.067585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-sys\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.067585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-host\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.067585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx62q\" (UniqueName: \"kubernetes.io/projected/d1e38884-a5a6-4a4e-9734-4415e7b28338-kube-api-access-zx62q\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.067585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-slash\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-env-overrides\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75434aec-7a98-4360-b03d-867428a3e8ae-ovn-node-metrics-cert\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-modprobe-d\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb8970da-0085-406e-b96f-8f18099e83a1-agent-certs\") pod \"konnectivity-agent-fpkj8\" (UID: \"fb8970da-0085-406e-b96f-8f18099e83a1\") " pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-slash\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49e4c0b3-6c4b-42ec-b83a-0df16108208a-iptables-alerter-script\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f86d9c73-bc49-4713-96dd-4b96e1cf9156-tmp-dir\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-var-lib-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-cni-netd\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-var-lib-kubelet\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-cni-bin\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-hostroot\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-cni-netd\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-tuned\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-var-lib-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.067976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.067973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qtk\" (UniqueName: \"kubernetes.io/projected/75434aec-7a98-4360-b03d-867428a3e8ae-kube-api-access-n4qtk\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9zz\" (UniqueName: \"kubernetes.io/projected/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-kube-api-access-4l9zz\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-cnibin\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-socket-dir-parent\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54rr\" (UniqueName: \"kubernetes.io/projected/048fbca3-d90e-4740-a1d0-1b7f77453e04-kube-api-access-k54rr\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.068132    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysctl-conf\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49e4c0b3-6c4b-42ec-b83a-0df16108208a-host-slash\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.068227    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret podName:99c997d6-cc19-4559-8591-58462280463f nodeName:}" failed. No retries permitted until 2026-09-09 21:38:48.568183812 +0000 UTC m=+2.990546507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret") pod "global-pull-secret-syncer-9kpfp" (UID: "99c997d6-cc19-4559-8591-58462280463f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49e4c0b3-6c4b-42ec-b83a-0df16108208a-host-slash\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f86d9c73-bc49-4713-96dd-4b96e1cf9156-tmp-dir\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-etc-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49e4c0b3-6c4b-42ec-b83a-0df16108208a-iptables-alerter-script\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-etc-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.068881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-device-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-openvswitch\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-log-socket\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-kubernetes\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-log-socket\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-systemd\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-systemd\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-env-overrides\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-cni-bin\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-cni-bin\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-lib-modules\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99c997d6-cc19-4559-8591-58462280463f-dbus\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-kubelet\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-system-cni-dir\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.069595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99c997d6-cc19-4559-8591-58462280463f-dbus\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-ovn\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-ovnkube-config\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-cnibin\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb8970da-0085-406e-b96f-8f18099e83a1-konnectivity-ca\") pod \"konnectivity-agent-fpkj8\" (UID: \"fb8970da-0085-406e-b96f-8f18099e83a1\") " pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-ovnkube-script-lib\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-system-cni-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-k8s-cni-cncf-io\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-etc-kubernetes\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068978    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.068996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-run-ovn\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkv2l\" (UniqueName: \"kubernetes.io/projected/49e4c0b3-6c4b-42ec-b83a-0df16108208a-kube-api-access-mkv2l\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-run-netns\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-netns\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-run-netns\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcldf\" (UniqueName: \"kubernetes.io/projected/d166fa40-a07a-4ca2-9909-37ca14b614b3-kube-api-access-lcldf\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-registration-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.070380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99c997d6-cc19-4559-8591-58462280463f-kubelet-config\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-ovnkube-script-lib\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4dg\" (UniqueName: \"kubernetes.io/projected/f86d9c73-bc49-4713-96dd-4b96e1cf9156-kube-api-access-tj4dg\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99c997d6-cc19-4559-8591-58462280463f-kubelet-config\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75434aec-7a98-4360-b03d-867428a3e8ae-ovnkube-config\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-systemd-units\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-node-log\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-systemd-units\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-serviceca\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-cni-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-node-log\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-multus-certs\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-socket-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-kubelet\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-conf-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-daemon-config\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75434aec-7a98-4360-b03d-867428a3e8ae-host-kubelet\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.070954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-cni-binary-copy\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysconfig\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysctl-d\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-host\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.069988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-cni-multus\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.070050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-sys-fs\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.070076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkr9\" (UniqueName: \"kubernetes.io/projected/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-kube-api-access-hbkr9\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.071528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.070112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-run\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.072527 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.072504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75434aec-7a98-4360-b03d-867428a3e8ae-ovn-node-metrics-cert\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.075984 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.075961    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:48.076243 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.075990    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:48.076243 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.076005    2579 projected.go:196] Error preparing data for projected volume kube-api-access-78hqx for pod openshift-network-diagnostics/network-check-target-zdcfg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:48.076243 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.076077    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx podName:e7e4dd6f-2775-469a-af85-3e5392ceaf31 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:48.576058553 +0000 UTC m=+2.998421254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-78hqx" (UniqueName: "kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx") pod "network-check-target-zdcfg" (UID: "e7e4dd6f-2775-469a-af85-3e5392ceaf31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:48.077481 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.077457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qtk\" (UniqueName: \"kubernetes.io/projected/75434aec-7a98-4360-b03d-867428a3e8ae-kube-api-access-n4qtk\") pod \"ovnkube-node-jjtjx\" (UID: \"75434aec-7a98-4360-b03d-867428a3e8ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.078774 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.078745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4dg\" (UniqueName: \"kubernetes.io/projected/f86d9c73-bc49-4713-96dd-4b96e1cf9156-kube-api-access-tj4dg\") pod \"node-resolver-vg5zh\" (UID: \"f86d9c73-bc49-4713-96dd-4b96e1cf9156\") " pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:48.079021 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.078996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkv2l\" (UniqueName: \"kubernetes.io/projected/49e4c0b3-6c4b-42ec-b83a-0df16108208a-kube-api-access-mkv2l\") pod \"iptables-alerter-fzqbr\" (UID: \"49e4c0b3-6c4b-42ec-b83a-0df16108208a\") " pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:48.090655 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.090589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal" event={"ID":"a1378741cd49bb53a4187c0b23b12755","Type":"ContainerStarted","Data":"e4ef62a0efdd599c8fc5a8afcf19d3ee3fe19dfb627303d74425496206475b9e"}
Sep 09 21:38:48.091921 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.091893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal" event={"ID":"c8ae9f8a9e6c826f807116759d8def20","Type":"ContainerStarted","Data":"e8a3ed4a4c4ac54d7261ba5ac87a10200f6f22766f9ab3f942124b5ed9a0dab6"}
Sep 09 21:38:48.170681 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.170846 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-var-lib-kubelet\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.170846 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-cni-bin\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.170846 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-cni-bin\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.170846 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-var-lib-kubelet\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.170846 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-hostroot\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-tuned\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-hostroot\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9zz\" (UniqueName: \"kubernetes.io/projected/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-kube-api-access-4l9zz\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-cnibin\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-socket-dir-parent\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.170994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54rr\" (UniqueName: \"kubernetes.io/projected/048fbca3-d90e-4740-a1d0-1b7f77453e04-kube-api-access-k54rr\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysctl-conf\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-cnibin\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-device-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.171105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-kubernetes\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-lib-modules\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-kubelet\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-system-cni-dir\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysctl-conf\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-cnibin\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb8970da-0085-406e-b96f-8f18099e83a1-konnectivity-ca\") pod \"konnectivity-agent-fpkj8\" (UID: \"fb8970da-0085-406e-b96f-8f18099e83a1\") " pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-system-cni-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-device-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-lib-modules\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-kubernetes\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-socket-dir-parent\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-cnibin\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-system-cni-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-system-cni-dir\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-kubelet\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-k8s-cni-cncf-io\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-k8s-cni-cncf-io\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.171693 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-etc-kubernetes\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-netns\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcldf\" (UniqueName: \"kubernetes.io/projected/d166fa40-a07a-4ca2-9909-37ca14b614b3-kube-api-access-lcldf\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-registration-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-serviceca\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-etc-kubernetes\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-cni-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-multus-certs\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-netns\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-socket-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-conf-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-daemon-config\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-cni-binary-copy\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysconfig\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-cni-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.172568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysctl-d\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-host\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-cni-multus\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb8970da-0085-406e-b96f-8f18099e83a1-konnectivity-ca\") pod \"konnectivity-agent-fpkj8\" (UID: \"fb8970da-0085-406e-b96f-8f18099e83a1\") " pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-sys-fs\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysctl-d\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-run-multus-certs\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-conf-dir\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.171994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-sys-fs\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-serviceca\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-host\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-socket-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-sysconfig\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-registration-dir\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkr9\" (UniqueName: \"kubernetes.io/projected/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-kube-api-access-hbkr9\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d166fa40-a07a-4ca2-9909-37ca14b614b3-host-var-lib-cni-multus\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.173495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8db25aa0-8917-4c9c-8d86-855d010e4c13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-run\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1e38884-a5a6-4a4e-9734-4415e7b28338-tmp\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172383    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-run\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d166fa40-a07a-4ca2-9909-37ca14b614b3-multus-daemon-config\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d166fa40-a07a-4ca2-9909-37ca14b614b3-cni-binary-copy\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-etc-selinux\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-cni-binary-copy\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hd8q\" (UniqueName: \"kubernetes.io/projected/8db25aa0-8917-4c9c-8d86-855d010e4c13-kube-api-access-5hd8q\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-etc-selinux\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-systemd\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-systemd\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-sys\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-host\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx62q\" (UniqueName: \"kubernetes.io/projected/d1e38884-a5a6-4a4e-9734-4415e7b28338-kube-api-access-zx62q\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-modprobe-d\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb8970da-0085-406e-b96f-8f18099e83a1-agent-certs\") pod \"konnectivity-agent-fpkj8\" (UID: \"fb8970da-0085-406e-b96f-8f18099e83a1\") " pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:48.174457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-host\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-modprobe-d\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e38884-a5a6-4a4e-9734-4415e7b28338-sys\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.172895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d166fa40-a07a-4ca2-9909-37ca14b614b3-cni-binary-copy\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.172970    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.173024    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:48.673006023 +0000 UTC m=+3.095368707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.173448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8db25aa0-8917-4c9c-8d86-855d010e4c13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.174500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1e38884-a5a6-4a4e-9734-4415e7b28338-etc-tuned\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.174861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1e38884-a5a6-4a4e-9734-4415e7b28338-tmp\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.175379 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.175242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb8970da-0085-406e-b96f-8f18099e83a1-agent-certs\") pod \"konnectivity-agent-fpkj8\" (UID: \"fb8970da-0085-406e-b96f-8f18099e83a1\") " pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:38:48.179976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.179950    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcldf\" (UniqueName: \"kubernetes.io/projected/d166fa40-a07a-4ca2-9909-37ca14b614b3-kube-api-access-lcldf\") pod \"multus-lqrz5\" (UID: \"d166fa40-a07a-4ca2-9909-37ca14b614b3\") " pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.180368 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.180340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9zz\" (UniqueName: \"kubernetes.io/projected/9bbe1e6f-07bc-4d52-b391-79178b7b6be9-kube-api-access-4l9zz\") pod \"node-ca-d8dsb\" (UID: \"9bbe1e6f-07bc-4d52-b391-79178b7b6be9\") " pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.180470 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.180408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hd8q\" (UniqueName: \"kubernetes.io/projected/8db25aa0-8917-4c9c-8d86-855d010e4c13-kube-api-access-5hd8q\") pod \"multus-additional-cni-plugins-7k8zp\" (UID: \"8db25aa0-8917-4c9c-8d86-855d010e4c13\") " pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.180850 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.180830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54rr\" (UniqueName: \"kubernetes.io/projected/048fbca3-d90e-4740-a1d0-1b7f77453e04-kube-api-access-k54rr\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:48.180957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.180848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkr9\" (UniqueName: \"kubernetes.io/projected/c5d382e6-fd63-4e1d-9adf-ccec64a57c87-kube-api-access-hbkr9\") pod \"aws-ebs-csi-driver-node-rm5jz\" (UID: \"c5d382e6-fd63-4e1d-9adf-ccec64a57c87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.183253 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.183228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx62q\" (UniqueName: \"kubernetes.io/projected/d1e38884-a5a6-4a4e-9734-4415e7b28338-kube-api-access-zx62q\") pod \"tuned-vvvk2\" (UID: \"d1e38884-a5a6-4a4e-9734-4415e7b28338\") " pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.259402 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.259361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:38:48.269799 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.269773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fzqbr"
Sep 09 21:38:48.278219 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.278198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vg5zh"
Sep 09 21:38:48.285817 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.285797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d8dsb"
Sep 09 21:38:48.293543 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.293517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lqrz5"
Sep 09 21:38:48.302401 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.302378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz"
Sep 09 21:38:48.309374 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.309354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvvk2"
Sep 09 21:38:48.316368 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.316347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7k8zp"
Sep 09 21:38:48.322018 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.321998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:38:48.398756 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.398724    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:48.575817 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.575739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:48.575960 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.575888    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:48.576015 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.575979    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret podName:99c997d6-cc19-4559-8591-58462280463f nodeName:}" failed. No retries permitted until 2026-09-09 21:38:49.575964101 +0000 UTC m=+3.998326778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret") pod "global-pull-secret-syncer-9kpfp" (UID: "99c997d6-cc19-4559-8591-58462280463f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:48.676168 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.676117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:48.676337 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:48.676186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:48.676337 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.676293    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:48.676337 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.676304    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:48.676337 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.676318    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:48.676337 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.676330    2579 projected.go:196] Error preparing data for projected volume kube-api-access-78hqx for pod openshift-network-diagnostics/network-check-target-zdcfg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:48.676577 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.676366    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:49.676346251 +0000 UTC m=+4.098708951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:48.676577 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:48.676380    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx podName:e7e4dd6f-2775-469a-af85-3e5392ceaf31 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:49.676374027 +0000 UTC m=+4.098736704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-78hqx" (UniqueName: "kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx") pod "network-check-target-zdcfg" (UID: "e7e4dd6f-2775-469a-af85-3e5392ceaf31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:48.787106 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.787075    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8970da_0085_406e_b96f_8f18099e83a1.slice/crio-fcf2c8105f50d06249243da2e69f876f6d4f2e9140a458160549c87d13ae4956 WatchSource:0}: Error finding container fcf2c8105f50d06249243da2e69f876f6d4f2e9140a458160549c87d13ae4956: Status 404 returned error can't find the container with id fcf2c8105f50d06249243da2e69f876f6d4f2e9140a458160549c87d13ae4956
Sep 09 21:38:48.787718 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.787695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e38884_a5a6_4a4e_9734_4415e7b28338.slice/crio-97c53ab83c54428e43751a27987eef56037b391a2f527093173f08812ae054cd WatchSource:0}: Error finding container 97c53ab83c54428e43751a27987eef56037b391a2f527093173f08812ae054cd: Status 404 returned error can't find the container with id 97c53ab83c54428e43751a27987eef56037b391a2f527093173f08812ae054cd
Sep 09 21:38:48.788582 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.788557    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d382e6_fd63_4e1d_9adf_ccec64a57c87.slice/crio-d32321baf4564b2260b70004652aaa4cb286b5a766cc154f6e218fcb199500cc WatchSource:0}: Error finding container d32321baf4564b2260b70004652aaa4cb286b5a766cc154f6e218fcb199500cc: Status 404 returned error can't find the container with id d32321baf4564b2260b70004652aaa4cb286b5a766cc154f6e218fcb199500cc
Sep 09 21:38:48.790063 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.789944    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75434aec_7a98_4360_b03d_867428a3e8ae.slice/crio-88ccf0dc789310d88b5c0b037737ed1e789b32c529c87a85c6a8fde689628997 WatchSource:0}: Error finding container 88ccf0dc789310d88b5c0b037737ed1e789b32c529c87a85c6a8fde689628997: Status 404 returned error can't find the container with id 88ccf0dc789310d88b5c0b037737ed1e789b32c529c87a85c6a8fde689628997
Sep 09 21:38:48.792868 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.792839    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd166fa40_a07a_4ca2_9909_37ca14b614b3.slice/crio-f3a5e8b2624bc508ea006aa9c368593c6942027f751c8c193a145bd3c2026f3b WatchSource:0}: Error finding container f3a5e8b2624bc508ea006aa9c368593c6942027f751c8c193a145bd3c2026f3b: Status 404 returned error can't find the container with id f3a5e8b2624bc508ea006aa9c368593c6942027f751c8c193a145bd3c2026f3b
Sep 09 21:38:48.795048 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.795024    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e4c0b3_6c4b_42ec_b83a_0df16108208a.slice/crio-78522c6aae02b8733a4a26a90d15171b4cc7ad5859e45a19d813a6c655214042 WatchSource:0}: Error finding container 78522c6aae02b8733a4a26a90d15171b4cc7ad5859e45a19d813a6c655214042: Status 404 returned error can't find the container with id 78522c6aae02b8733a4a26a90d15171b4cc7ad5859e45a19d813a6c655214042
Sep 09 21:38:48.814166 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.814144    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86d9c73_bc49_4713_96dd_4b96e1cf9156.slice/crio-7b1ac736b5f9a225686ac4dece3ba42fa4e1ae595569f266cfaad15b9604cf60 WatchSource:0}: Error finding container 7b1ac736b5f9a225686ac4dece3ba42fa4e1ae595569f266cfaad15b9604cf60: Status 404 returned error can't find the container with id 7b1ac736b5f9a225686ac4dece3ba42fa4e1ae595569f266cfaad15b9604cf60
Sep 09 21:38:48.814936 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.814914    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbe1e6f_07bc_4d52_b391_79178b7b6be9.slice/crio-7b69084ab5d634ec78d36bb8fd2cc4a4d46bb26b99f787d98a80abfc4e7ac221 WatchSource:0}: Error finding container 7b69084ab5d634ec78d36bb8fd2cc4a4d46bb26b99f787d98a80abfc4e7ac221: Status 404 returned error can't find the container with id 7b69084ab5d634ec78d36bb8fd2cc4a4d46bb26b99f787d98a80abfc4e7ac221
Sep 09 21:38:48.816110 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:38:48.816071    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db25aa0_8917_4c9c_8d86_855d010e4c13.slice/crio-3ac204fb1d06ff5764a4a4ada698769872673ee0033a9fc7b6bbc7d658ed8c0a WatchSource:0}: Error finding container 3ac204fb1d06ff5764a4a4ada698769872673ee0033a9fc7b6bbc7d658ed8c0a: Status 404 returned error can't find the container with id 3ac204fb1d06ff5764a4a4ada698769872673ee0033a9fc7b6bbc7d658ed8c0a
Sep 09 21:38:49.001854 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.001664    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 21:33:47 +0000 UTC" deadline="2028-05-26 17:04:11.037153628 +0000 UTC"
Sep 09 21:38:49.001854 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.001850    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14995h25m22.035309269s"
Sep 09 21:38:49.086446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.086369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:49.086623 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.086483    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:38:49.095806 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.095754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz" event={"ID":"c5d382e6-fd63-4e1d-9adf-ccec64a57c87","Type":"ContainerStarted","Data":"d32321baf4564b2260b70004652aaa4cb286b5a766cc154f6e218fcb199500cc"}
Sep 09 21:38:49.098352 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.098287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal" event={"ID":"a1378741cd49bb53a4187c0b23b12755","Type":"ContainerStarted","Data":"2f5e07533d19cfd33e6586871f04d04ae689c5a4b74523e18ee58bceb9b5dc48"}
Sep 09 21:38:49.099672 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.099640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d8dsb" event={"ID":"9bbe1e6f-07bc-4d52-b391-79178b7b6be9","Type":"ContainerStarted","Data":"7b69084ab5d634ec78d36bb8fd2cc4a4d46bb26b99f787d98a80abfc4e7ac221"}
Sep 09 21:38:49.100922 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.100867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvvk2" event={"ID":"d1e38884-a5a6-4a4e-9734-4415e7b28338","Type":"ContainerStarted","Data":"97c53ab83c54428e43751a27987eef56037b391a2f527093173f08812ae054cd"}
Sep 09 21:38:49.102266 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.102240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fpkj8" event={"ID":"fb8970da-0085-406e-b96f-8f18099e83a1","Type":"ContainerStarted","Data":"fcf2c8105f50d06249243da2e69f876f6d4f2e9140a458160549c87d13ae4956"}
Sep 09 21:38:49.103654 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.103630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" event={"ID":"8db25aa0-8917-4c9c-8d86-855d010e4c13","Type":"ContainerStarted","Data":"3ac204fb1d06ff5764a4a4ada698769872673ee0033a9fc7b6bbc7d658ed8c0a"}
Sep 09 21:38:49.104860 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.104833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vg5zh" event={"ID":"f86d9c73-bc49-4713-96dd-4b96e1cf9156","Type":"ContainerStarted","Data":"7b1ac736b5f9a225686ac4dece3ba42fa4e1ae595569f266cfaad15b9604cf60"}
Sep 09 21:38:49.106269 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.106244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fzqbr" event={"ID":"49e4c0b3-6c4b-42ec-b83a-0df16108208a","Type":"ContainerStarted","Data":"78522c6aae02b8733a4a26a90d15171b4cc7ad5859e45a19d813a6c655214042"}
Sep 09 21:38:49.107633 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.107571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lqrz5" event={"ID":"d166fa40-a07a-4ca2-9909-37ca14b614b3","Type":"ContainerStarted","Data":"f3a5e8b2624bc508ea006aa9c368593c6942027f751c8c193a145bd3c2026f3b"}
Sep 09 21:38:49.108802 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.108780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"88ccf0dc789310d88b5c0b037737ed1e789b32c529c87a85c6a8fde689628997"}
Sep 09 21:38:49.112026 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.111973    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-221.ec2.internal" podStartSLOduration=2.111961371 podStartE2EDuration="2.111961371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:49.111503031 +0000 UTC m=+3.533865726" watchObservedRunningTime="2026-09-09 21:38:49.111961371 +0000 UTC m=+3.534324067"
Sep 09 21:38:49.582964 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.582871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:49.583128 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.583058    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:49.583128 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.583126    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret podName:99c997d6-cc19-4559-8591-58462280463f nodeName:}" failed. No retries permitted until 2026-09-09 21:38:51.583107161 +0000 UTC m=+6.005469842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret") pod "global-pull-secret-syncer-9kpfp" (UID: "99c997d6-cc19-4559-8591-58462280463f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:49.683682 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.683616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:49.683885 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.683701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:49.683960 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.683883    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:49.683960 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.683910    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:49.683960 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.683923    2579 projected.go:196] Error preparing data for projected volume kube-api-access-78hqx for pod openshift-network-diagnostics/network-check-target-zdcfg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:49.684113 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.683982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx podName:e7e4dd6f-2775-469a-af85-3e5392ceaf31 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:51.683964803 +0000 UTC m=+6.106327484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-78hqx" (UniqueName: "kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx") pod "network-check-target-zdcfg" (UID: "e7e4dd6f-2775-469a-af85-3e5392ceaf31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:49.684529 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.684475    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:49.684648 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:49.684534    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:51.684519288 +0000 UTC m=+6.106881991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:49.809273 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:49.809216    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 21:38:50.002997 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:50.002873    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 21:33:47 +0000 UTC" deadline="2028-05-28 20:55:41.77481378 +0000 UTC"
Sep 09 21:38:50.002997 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:50.002950    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15047h16m51.771868352s"
Sep 09 21:38:50.086758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:50.086724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:50.086922 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:50.086866    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:38:50.086997 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:50.086948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:50.087051 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:50.087034    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:38:51.086169 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:51.086126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:51.086915 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.086251    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:38:51.129283 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:51.128321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal" event={"ID":"c8ae9f8a9e6c826f807116759d8def20","Type":"ContainerStarted","Data":"b1af070c21d49d52be303625186054c3da643adef2a00884af06a1353a3ebba2"}
Sep 09 21:38:51.602970 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:51.602929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:51.603152 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.603081    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:51.603152 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.603146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret podName:99c997d6-cc19-4559-8591-58462280463f nodeName:}" failed. No retries permitted until 2026-09-09 21:38:55.603128088 +0000 UTC m=+10.025490780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret") pod "global-pull-secret-syncer-9kpfp" (UID: "99c997d6-cc19-4559-8591-58462280463f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:51.703868 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:51.703821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:51.704036 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:51.703906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:51.704106 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.704066    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:51.704106 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.704087    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:51.704106 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.704100    2579 projected.go:196] Error preparing data for projected volume kube-api-access-78hqx for pod openshift-network-diagnostics/network-check-target-zdcfg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:51.704265 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.704159    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx podName:e7e4dd6f-2775-469a-af85-3e5392ceaf31 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:55.70414154 +0000 UTC m=+10.126504235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-78hqx" (UniqueName: "kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx") pod "network-check-target-zdcfg" (UID: "e7e4dd6f-2775-469a-af85-3e5392ceaf31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:51.704265 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.704245    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:51.704372 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:51.704295    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:38:55.704281836 +0000 UTC m=+10.126644529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:52.087787 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:52.086731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:52.087787 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:52.086914    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:38:52.090652 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:52.088387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:52.090652 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:52.088497    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:38:53.086448 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:53.086411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:53.086662 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:53.086558    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:38:54.087637 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:54.087135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:54.087637 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:54.087180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:54.087637 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:54.087279    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:38:54.087637 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:54.087446    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:38:55.086353 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:55.086262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:55.086496 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.086403    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:38:55.637261 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:55.637184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:55.637743 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.637342    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:55.637743 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.637421    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret podName:99c997d6-cc19-4559-8591-58462280463f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:03.637398309 +0000 UTC m=+18.059760989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret") pod "global-pull-secret-syncer-9kpfp" (UID: "99c997d6-cc19-4559-8591-58462280463f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:38:55.737811 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:55.737658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:55.737811 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:55.737743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:55.738058 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.737890    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:55.738058 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.737953    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:03.737934641 +0000 UTC m=+18.160297322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:38:55.738409 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.738384    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:38:55.738501 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.738414    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:38:55.738501 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.738428    2579 projected.go:196] Error preparing data for projected volume kube-api-access-78hqx for pod openshift-network-diagnostics/network-check-target-zdcfg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:55.738501 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:55.738478    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx podName:e7e4dd6f-2775-469a-af85-3e5392ceaf31 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:03.738461183 +0000 UTC m=+18.160823863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-78hqx" (UniqueName: "kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx") pod "network-check-target-zdcfg" (UID: "e7e4dd6f-2775-469a-af85-3e5392ceaf31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:38:56.087635 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:56.087172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:56.087635 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:56.087280    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:38:56.087866 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:56.087665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:56.087866 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:56.087770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:38:57.086065 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:57.085516    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:57.086065 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:57.085667    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:38:57.146830 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:57.146783    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8ae9f8a9e6c826f807116759d8def20" containerID="b1af070c21d49d52be303625186054c3da643adef2a00884af06a1353a3ebba2" exitCode=0
Sep 09 21:38:57.147018 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:57.146837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal" event={"ID":"c8ae9f8a9e6c826f807116759d8def20","Type":"ContainerDied","Data":"b1af070c21d49d52be303625186054c3da643adef2a00884af06a1353a3ebba2"}
Sep 09 21:38:58.085910 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:58.085874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:38:58.086094 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:58.086005    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:38:58.086094 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:58.086061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:38:58.086437 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:58.086190    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:38:59.086398 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:38:59.086362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:38:59.086785 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:38:59.086493    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:00.085819 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:00.085779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:00.085996 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:00.085792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:00.085996 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:00.085912    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:00.086099 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:00.086010    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:01.085727 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:01.085701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:01.086161 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:01.085813    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:02.085703 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:02.085667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:02.085893 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:02.085715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:02.085893 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:02.085822    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:02.086388 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:02.085974    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:03.085996 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:03.085952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:03.086444 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.086078    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:03.695435 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:03.695397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:03.695649 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.695578    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:39:03.695710 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.695660    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret podName:99c997d6-cc19-4559-8591-58462280463f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:19.695639667 +0000 UTC m=+34.118002365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret") pod "global-pull-secret-syncer-9kpfp" (UID: "99c997d6-cc19-4559-8591-58462280463f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:39:03.796291 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:03.796248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:03.796467 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:03.796312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:03.796467 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.796404    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:39:03.796467 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.796433    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:39:03.796467 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.796465    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:39:03.796670 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.796471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:19.796454027 +0000 UTC m=+34.218816706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:39:03.796670 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.796478    2579 projected.go:196] Error preparing data for projected volume kube-api-access-78hqx for pod openshift-network-diagnostics/network-check-target-zdcfg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:39:03.796670 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:03.796527    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx podName:e7e4dd6f-2775-469a-af85-3e5392ceaf31 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:19.796511379 +0000 UTC m=+34.218874056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-78hqx" (UniqueName: "kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx") pod "network-check-target-zdcfg" (UID: "e7e4dd6f-2775-469a-af85-3e5392ceaf31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:39:04.086065 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:04.085974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:04.086510 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:04.085985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:04.086510 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:04.086123    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:04.086510 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:04.086194    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:05.086035 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:05.085997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:05.086238 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:05.086133    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:06.086829 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:06.086786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:06.087306 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:06.086924    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:06.087306 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:06.086976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:06.087306 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:06.087099    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:07.086104 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.085797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:07.086256 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:07.086177    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:07.166880 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.166765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz" event={"ID":"c5d382e6-fd63-4e1d-9adf-ccec64a57c87","Type":"ContainerStarted","Data":"14902a2f99f6652381eebf8e79bc519b814a762b50b391b2389cb14440c62a7f"}
Sep 09 21:39:07.169011 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.168966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal" event={"ID":"c8ae9f8a9e6c826f807116759d8def20","Type":"ContainerStarted","Data":"70b0862f9d62784415bb1139bb6541c3c947ee9ce6b558676dfc79eeeab361b6"}
Sep 09 21:39:07.170684 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.170646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d8dsb" event={"ID":"9bbe1e6f-07bc-4d52-b391-79178b7b6be9","Type":"ContainerStarted","Data":"002e250164fcc0e8eb2722cf68cb1dba4ce7d4059e923f26bc60f56c19c48629"}
Sep 09 21:39:07.172329 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.172301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvvk2" event={"ID":"d1e38884-a5a6-4a4e-9734-4415e7b28338","Type":"ContainerStarted","Data":"d3857e5b4c55b37e86179eef13f812205c1bbc80d4f5456c4685bc5917c47dc5"}
Sep 09 21:39:07.174044 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.174019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fpkj8" event={"ID":"fb8970da-0085-406e-b96f-8f18099e83a1","Type":"ContainerStarted","Data":"9bfc815a22774935229760646df593aeb47b0855a454a49ca4cb2d0f0f73c18a"}
Sep 09 21:39:07.175917 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.175882    2579 generic.go:356] "Generic (PLEG): container finished" podID="8db25aa0-8917-4c9c-8d86-855d010e4c13" containerID="5c2403c9ed7342373b7745bfac22f72ee4c2444d1a9c3abc6cc93b291a7c9286" exitCode=0
Sep 09 21:39:07.176030 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.175939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" event={"ID":"8db25aa0-8917-4c9c-8d86-855d010e4c13","Type":"ContainerDied","Data":"5c2403c9ed7342373b7745bfac22f72ee4c2444d1a9c3abc6cc93b291a7c9286"}
Sep 09 21:39:07.177713 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.177666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vg5zh" event={"ID":"f86d9c73-bc49-4713-96dd-4b96e1cf9156","Type":"ContainerStarted","Data":"5a599285090094f191b27f7fe734ad710da397a14b76756f4b2d7fae211d7926"}
Sep 09 21:39:07.179669 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.179642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lqrz5" event={"ID":"d166fa40-a07a-4ca2-9909-37ca14b614b3","Type":"ContainerStarted","Data":"dc707ab2a151762872f02efcad68e939de3410e4c1cdcf1d07f3c36cf6294d90"}
Sep 09 21:39:07.182787 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.182748    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-221.ec2.internal" podStartSLOduration=20.18273232 podStartE2EDuration="20.18273232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:07.18224595 +0000 UTC m=+21.604608661" watchObservedRunningTime="2026-09-09 21:39:07.18273232 +0000 UTC m=+21.605095020"
Sep 09 21:39:07.183318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.183291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"9740d5c1b8ca9f9df6cafdc078bb70fb0e42ffe77f8ef9ea433b85fd10097f68"}
Sep 09 21:39:07.183424 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.183365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"cfc2e62ce91234de0f91b5d0fbae69fb2678fb24d9a32610931b0e284d319303"}
Sep 09 21:39:07.183424 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.183415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"70e54adad8271e5494e75b2484c75573a18a2c4cd3bef7c5eb7e7f76b077dd76"}
Sep 09 21:39:07.183534 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.183464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"f7c6b43ae2a0f5bdd156f4fb41cd7daa54193c0c33950bceef4e638edaa42377"}
Sep 09 21:39:07.183534 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.183513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"a98970b64f3ccdbcc7a3e9219192c14d40d7e3aa000d07eacb013db3d0c2f91a"}
Sep 09 21:39:07.183646 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.183560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"4d70566f276de7d83ac5689c90413a82e261c2390ff76c7173c2120ef2a8ebce"}
Sep 09 21:39:07.200758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.200696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vg5zh" podStartSLOduration=3.560611652 podStartE2EDuration="21.20068082s" totalImagesPullingTime="17.640069168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:07.200058616 +0000 UTC m=+21.622421318" watchObservedRunningTime="2026-09-09 21:39:07.20068082 +0000 UTC m=+21.623043522"
Sep 09 21:39:07.213986 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.213910    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d8dsb" podStartSLOduration=3.573945359 podStartE2EDuration="21.21389145s" totalImagesPullingTime="17.639946091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:07.213548436 +0000 UTC m=+21.635911137" watchObservedRunningTime="2026-09-09 21:39:07.21389145 +0000 UTC m=+21.636254151"
Sep 09 21:39:07.229486 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.229444    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lqrz5" podStartSLOduration=3.547513163 podStartE2EDuration="21.229426904s" totalImagesPullingTime="17.681913741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:07.229026239 +0000 UTC m=+21.651388940" watchObservedRunningTime="2026-09-09 21:39:07.229426904 +0000 UTC m=+21.651789606"
Sep 09 21:39:07.245856 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.245812    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vvvk2" podStartSLOduration=3.57593266 podStartE2EDuration="21.245795935s" totalImagesPullingTime="17.669863275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:07.245736532 +0000 UTC m=+21.668099232" watchObservedRunningTime="2026-09-09 21:39:07.245795935 +0000 UTC m=+21.668158637"
Sep 09 21:39:07.276827 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.276787    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fpkj8" podStartSLOduration=7.689386703 podStartE2EDuration="21.276774475s" totalImagesPullingTime="13.587387772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:07.276667027 +0000 UTC m=+21.699029727" watchObservedRunningTime="2026-09-09 21:39:07.276774475 +0000 UTC m=+21.699137174"
Sep 09 21:39:07.723758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:07.723737    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 21:39:08.030766 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:08.030671    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T21:39:07.723753892Z","UUID":"8a15b53d-5656-4b7f-aa41-731baea1109c","Handler":null,"Name":"","Endpoint":""}
Sep 09 21:39:08.033017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:08.032771    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 21:39:08.033017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:08.032807    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 21:39:08.089360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:08.089285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:08.089360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:08.089311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:08.089578 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:08.089424    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:08.089578 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:08.089526    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:08.187522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:08.187473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fzqbr" event={"ID":"49e4c0b3-6c4b-42ec-b83a-0df16108208a","Type":"ContainerStarted","Data":"8eab4bbca1d82c8db4f27d2049d2eb25c77a7fda156b31a459c15a42804b3d45"}
Sep 09 21:39:08.189770 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:08.189724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz" event={"ID":"c5d382e6-fd63-4e1d-9adf-ccec64a57c87","Type":"ContainerStarted","Data":"2ec113ee7f6d0a2dd52b8ae4d9fd1b993facb5b603e2eca1190a1a793c6f94a0"}
Sep 09 21:39:08.202381 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:08.202310    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fzqbr" podStartSLOduration=4.556300567 podStartE2EDuration="22.202290273s" totalImagesPullingTime="17.645989706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:08.201619824 +0000 UTC m=+22.623982522" watchObservedRunningTime="2026-09-09 21:39:08.202290273 +0000 UTC m=+22.624653020"
Sep 09 21:39:09.086469 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:09.086443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:09.086590 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:09.086568    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:09.197164 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:09.197113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"5475c8091c80d273c207671016c9c1f73a04cdd8d89b69c95b2a2c170bb39264"}
Sep 09 21:39:09.999868 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:09.999831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:39:10.089153 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:10.089070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:10.089284 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:10.089181    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:10.089284 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:10.089228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:10.089354 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:10.089324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:10.201853 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:10.201818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz" event={"ID":"c5d382e6-fd63-4e1d-9adf-ccec64a57c87","Type":"ContainerStarted","Data":"daa1de3791d8dd000731dc81c26f06a56759e3890122ece6a07c150c9cd39bb7"}
Sep 09 21:39:11.086039 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:11.085844    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:11.086232 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:11.086137    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:11.981620 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:11.981568    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:39:11.982291 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:11.982269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:39:11.996420 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:11.996369    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rm5jz" podStartSLOduration=5.322725288 podStartE2EDuration="25.996349155s" totalImagesPullingTime="20.673623867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:10.244616906 +0000 UTC m=+24.666979598" watchObservedRunningTime="2026-09-09 21:39:11.996349155 +0000 UTC m=+26.418711857"
Sep 09 21:39:12.090242 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:12.090208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:12.090242 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:12.090231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:12.090452 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:12.090354    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:12.090502 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:12.090476    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:12.206795 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:12.206766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fpkj8"
Sep 09 21:39:13.086333 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:13.086152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:13.086726 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:13.086412    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:13.209497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:13.209399    2579 generic.go:356] "Generic (PLEG): container finished" podID="8db25aa0-8917-4c9c-8d86-855d010e4c13" containerID="a156a1181367af9e039281855c09bc776a6772e58b292fa2fbcc214a06fd0997" exitCode=0
Sep 09 21:39:13.209497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:13.209454    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" event={"ID":"8db25aa0-8917-4c9c-8d86-855d010e4c13","Type":"ContainerDied","Data":"a156a1181367af9e039281855c09bc776a6772e58b292fa2fbcc214a06fd0997"}
Sep 09 21:39:13.213655 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:13.213624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" event={"ID":"75434aec-7a98-4360-b03d-867428a3e8ae","Type":"ContainerStarted","Data":"be1ddda3ec43e6e8923f313584f502df44a854296081c51c151e30d27d3ddf1a"}
Sep 09 21:39:13.253382 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:13.253332    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx" podStartSLOduration=9.451116057 podStartE2EDuration="27.253317231s" totalImagesPullingTime="17.802201174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:13.251856614 +0000 UTC m=+27.674219313" watchObservedRunningTime="2026-09-09 21:39:13.253317231 +0000 UTC m=+27.675679931"
Sep 09 21:39:14.086580 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.086549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:14.087161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.086535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:14.087161 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:14.086699    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:14.087161 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:14.086756    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:14.217996 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.217970    2579 generic.go:356] "Generic (PLEG): container finished" podID="8db25aa0-8917-4c9c-8d86-855d010e4c13" containerID="1fdec28b30972e62e4f4d7af10b5a1bd14a858ea8540b574124b154f283105ed" exitCode=0
Sep 09 21:39:14.218099 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.218057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" event={"ID":"8db25aa0-8917-4c9c-8d86-855d010e4c13","Type":"ContainerDied","Data":"1fdec28b30972e62e4f4d7af10b5a1bd14a858ea8540b574124b154f283105ed"}
Sep 09 21:39:14.218472 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.218447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:39:14.218577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.218516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:39:14.218577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.218565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:39:14.232967 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.232946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:39:14.233058 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.233038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:39:14.323415 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.323319    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zdcfg"]
Sep 09 21:39:14.323562 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.323502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:14.323680 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:14.323654    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:14.323900 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.323880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9kpfp"]
Sep 09 21:39:14.323970 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.323956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:14.324059 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:14.324033    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:14.325860 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.325837    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wvv59"]
Sep 09 21:39:14.325940 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:14.325917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:14.326018 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:14.325993    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:15.223499 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:15.223464    2579 generic.go:356] "Generic (PLEG): container finished" podID="8db25aa0-8917-4c9c-8d86-855d010e4c13" containerID="c61896ad196627edb3e9e3aff4fa4dd31cad1113e66b61f6c4e6f9c034bb55a5" exitCode=0
Sep 09 21:39:15.224115 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:15.223505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" event={"ID":"8db25aa0-8917-4c9c-8d86-855d010e4c13","Type":"ContainerDied","Data":"c61896ad196627edb3e9e3aff4fa4dd31cad1113e66b61f6c4e6f9c034bb55a5"}
Sep 09 21:39:16.089237 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:16.089013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:16.089421 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:16.089085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:16.089421 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:16.089315    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:16.089421 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:16.089113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:16.089421 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:16.089403    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:16.089665 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:16.089499    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:18.086391 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:18.086364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:18.086768 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:18.086365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:18.086768 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:18.086463    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zdcfg" podUID="e7e4dd6f-2775-469a-af85-3e5392ceaf31"
Sep 09 21:39:18.086768 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:18.086576    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:39:18.086768 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:18.086365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:18.086768 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:18.086706    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kpfp" podUID="99c997d6-cc19-4559-8591-58462280463f"
Sep 09 21:39:19.373009 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.372975    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-221.ec2.internal" event="NodeReady"
Sep 09 21:39:19.373478 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.373137    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 21:39:19.413892 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.413860    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v99wb"]
Sep 09 21:39:19.502400 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.502353    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mfrlp"]
Sep 09 21:39:19.502582 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.502453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.504860 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.504831    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 21:39:19.504966 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.504894    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8h77q\""
Sep 09 21:39:19.505034 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.504832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 21:39:19.527810 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.527775    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v99wb"]
Sep 09 21:39:19.527991 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.527828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfrlp"
Sep 09 21:39:19.527991 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.527843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfrlp"]
Sep 09 21:39:19.530752 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.530714    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bllts\""
Sep 09 21:39:19.530752 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.530726    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 21:39:19.530752 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.530741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 21:39:19.531055 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.530888    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 21:39:19.608984 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.608947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnkp\" (UniqueName: \"kubernetes.io/projected/667ff566-22cb-4cdc-beb3-6bab86102ec5-kube-api-access-5dnkp\") pod \"ingress-canary-mfrlp\" (UID: \"667ff566-22cb-4cdc-beb3-6bab86102ec5\") " pod="openshift-ingress-canary/ingress-canary-mfrlp"
Sep 09 21:39:19.608984 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.608989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d834a708-778c-44a5-ad5a-ecdcb56aa53f-tmp-dir\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.609244 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.609020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/667ff566-22cb-4cdc-beb3-6bab86102ec5-cert\") pod \"ingress-canary-mfrlp\" (UID: \"667ff566-22cb-4cdc-beb3-6bab86102ec5\") " pod="openshift-ingress-canary/ingress-canary-mfrlp"
Sep 09 21:39:19.609244 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.609134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d834a708-778c-44a5-ad5a-ecdcb56aa53f-config-volume\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.609244 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.609195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.609385 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.609225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvbq\" (UniqueName: \"kubernetes.io/projected/d834a708-778c-44a5-ad5a-ecdcb56aa53f-kube-api-access-ljvbq\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.710328 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.710242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d834a708-778c-44a5-ad5a-ecdcb56aa53f-config-volume\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.710328 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.710294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.710328 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.710321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvbq\" (UniqueName: \"kubernetes.io/projected/d834a708-778c-44a5-ad5a-ecdcb56aa53f-kube-api-access-ljvbq\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.710572 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.710440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnkp\" (UniqueName: \"kubernetes.io/projected/667ff566-22cb-4cdc-beb3-6bab86102ec5-kube-api-access-5dnkp\") pod \"ingress-canary-mfrlp\" (UID: \"667ff566-22cb-4cdc-beb3-6bab86102ec5\") " pod="openshift-ingress-canary/ingress-canary-mfrlp"
Sep 09 21:39:19.710572 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.710448    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:19.710572 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.710469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d834a708-778c-44a5-ad5a-ecdcb56aa53f-tmp-dir\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.710572 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.710534    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:20.210511913 +0000 UTC m=+34.632874606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:19.710801 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.710673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/667ff566-22cb-4cdc-beb3-6bab86102ec5-cert\") pod \"ingress-canary-mfrlp\" (UID: \"667ff566-22cb-4cdc-beb3-6bab86102ec5\") " pod="openshift-ingress-canary/ingress-canary-mfrlp"
Sep 09 21:39:19.710801 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.710724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:19.710900 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.710842    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 21:39:19.710951 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.710914    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret podName:99c997d6-cc19-4559-8591-58462280463f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:51.71089715 +0000 UTC m=+66.133259841 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret") pod "global-pull-secret-syncer-9kpfp" (UID: "99c997d6-cc19-4559-8591-58462280463f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 21:39:19.711207 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.711180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d834a708-778c-44a5-ad5a-ecdcb56aa53f-tmp-dir\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.711374 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.711350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d834a708-778c-44a5-ad5a-ecdcb56aa53f-config-volume\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.715541 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.715518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/667ff566-22cb-4cdc-beb3-6bab86102ec5-cert\") pod \"ingress-canary-mfrlp\" (UID: \"667ff566-22cb-4cdc-beb3-6bab86102ec5\") " pod="openshift-ingress-canary/ingress-canary-mfrlp"
Sep 09 21:39:19.719158 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.719136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvbq\" (UniqueName: \"kubernetes.io/projected/d834a708-778c-44a5-ad5a-ecdcb56aa53f-kube-api-access-ljvbq\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:19.719495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.719425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnkp\" (UniqueName: \"kubernetes.io/projected/667ff566-22cb-4cdc-beb3-6bab86102ec5-kube-api-access-5dnkp\") pod \"ingress-canary-mfrlp\" (UID: \"667ff566-22cb-4cdc-beb3-6bab86102ec5\") " pod="openshift-ingress-canary/ingress-canary-mfrlp"
Sep 09 21:39:19.811699 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.811662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:19.811874 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.811724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:19.811874 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.811829    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:39:19.811874 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.811846    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 21:39:19.811874 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.811868    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 21:39:19.812090 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.811881    2579 projected.go:196] Error preparing data for projected volume kube-api-access-78hqx for pod openshift-network-diagnostics/network-check-target-zdcfg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:39:19.812090 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.811908    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:51.811887957 +0000 UTC m=+66.234250654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 21:39:19.812090 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:19.811942    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx podName:e7e4dd6f-2775-469a-af85-3e5392ceaf31 nodeName:}" failed. No retries permitted until 2026-09-09 21:39:51.811929795 +0000 UTC m=+66.234292487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-78hqx" (UniqueName: "kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx") pod "network-check-target-zdcfg" (UID: "e7e4dd6f-2775-469a-af85-3e5392ceaf31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 21:39:19.838519 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:19.838486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfrlp"
Sep 09 21:39:20.085701 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.085669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:20.085701 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.085705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:20.085930 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.085669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:20.088364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.088339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 21:39:20.088364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.088365    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 21:39:20.088364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.088372    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 21:39:20.088685 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.088513    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-znm4z\""
Sep 09 21:39:20.088685 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.088681    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 21:39:20.089278 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.089258    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6js7k\""
Sep 09 21:39:20.207775 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.207746    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfrlp"]
Sep 09 21:39:20.216319 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:20.216290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:20.216425 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:20.216410    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:20.216492 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:20.216483    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:21.216464204 +0000 UTC m=+35.638827110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:20.247267 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:39:20.247198    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667ff566_22cb_4cdc_beb3_6bab86102ec5.slice/crio-c72037a558ffee4cbbf653d148e332531aa6728f1f1f8ba8fa15f814a1505db4 WatchSource:0}: Error finding container c72037a558ffee4cbbf653d148e332531aa6728f1f1f8ba8fa15f814a1505db4: Status 404 returned error can't find the container with id c72037a558ffee4cbbf653d148e332531aa6728f1f1f8ba8fa15f814a1505db4
Sep 09 21:39:21.223861 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:21.223649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:21.224319 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:21.223840    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:21.224319 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:21.223951    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:23.223935324 +0000 UTC m=+37.646298003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:21.239664 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:21.239630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfrlp" event={"ID":"667ff566-22cb-4cdc-beb3-6bab86102ec5","Type":"ContainerStarted","Data":"c72037a558ffee4cbbf653d148e332531aa6728f1f1f8ba8fa15f814a1505db4"}
Sep 09 21:39:21.242775 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:21.242745    2579 generic.go:356] "Generic (PLEG): container finished" podID="8db25aa0-8917-4c9c-8d86-855d010e4c13" containerID="5489476fa10f25f2d944051a4b14eb5ab41e6069a4dc69c470df6b40d50e99a2" exitCode=0
Sep 09 21:39:21.242899 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:21.242787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" event={"ID":"8db25aa0-8917-4c9c-8d86-855d010e4c13","Type":"ContainerDied","Data":"5489476fa10f25f2d944051a4b14eb5ab41e6069a4dc69c470df6b40d50e99a2"}
Sep 09 21:39:22.249415 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:22.249377    2579 generic.go:356] "Generic (PLEG): container finished" podID="8db25aa0-8917-4c9c-8d86-855d010e4c13" containerID="90a6b306c1d0aaf097323fe60e88f64e2d5be0aaca2250d826807d2f4b492d68" exitCode=0
Sep 09 21:39:22.250038 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:22.249452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" event={"ID":"8db25aa0-8917-4c9c-8d86-855d010e4c13","Type":"ContainerDied","Data":"90a6b306c1d0aaf097323fe60e88f64e2d5be0aaca2250d826807d2f4b492d68"}
Sep 09 21:39:23.238397 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:23.238310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:23.238536 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:23.238464    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:23.238536 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:23.238531    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:27.238512219 +0000 UTC m=+41.660874920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:23.255382 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:23.255345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" event={"ID":"8db25aa0-8917-4c9c-8d86-855d010e4c13","Type":"ContainerStarted","Data":"f43a6654d7fa45ee02a60c63b3a72244eeeeec6fd957cc2624be48995dc0ca99"}
Sep 09 21:39:23.256906 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:23.256880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfrlp" event={"ID":"667ff566-22cb-4cdc-beb3-6bab86102ec5","Type":"ContainerStarted","Data":"c1b580a9058cbc804ffaf431fa76cd17697119fe4ae28791433e4803f08dc54c"}
Sep 09 21:39:23.277443 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:23.277391    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7k8zp" podStartSLOduration=7.710818457 podStartE2EDuration="37.277375504s" totalImagesPullingTime="29.566557047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:23.275762784 +0000 UTC m=+37.698125484" watchObservedRunningTime="2026-09-09 21:39:23.277375504 +0000 UTC m=+37.699738204"
Sep 09 21:39:23.290438 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:23.290377    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mfrlp" podStartSLOduration=1.723431009 podStartE2EDuration="4.29036123s" totalImagesPullingTime="2.566930221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:23.290284627 +0000 UTC m=+37.712647327" watchObservedRunningTime="2026-09-09 21:39:23.29036123 +0000 UTC m=+37.712723928"
Sep 09 21:39:27.267659 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:27.267613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:27.268068 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:27.267700    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:27.268068 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:27.267756    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:35.267741167 +0000 UTC m=+49.690103850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:35.323490 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:35.323446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:35.323945 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:35.323594    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:35.323945 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:35.323672    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:39:51.323656131 +0000 UTC m=+65.746018812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:46.236253 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:46.236224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjtjx"
Sep 09 21:39:51.336358 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.336312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:39:51.336875 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:51.336463    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:39:51.336875 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:51.336539    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:40:23.336523062 +0000 UTC m=+97.758885755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:39:51.738832 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.738791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:51.741331 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.741314    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 21:39:51.752085 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.752064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c997d6-cc19-4559-8591-58462280463f-original-pull-secret\") pod \"global-pull-secret-syncer-9kpfp\" (UID: \"99c997d6-cc19-4559-8591-58462280463f\") " pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:51.839804 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.839769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:51.839925 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.839828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:39:51.842312 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.842294    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 21:39:51.842442 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.842424    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 21:39:51.850368 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:51.850343    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 21:39:51.850455 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:39:51.850412    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:40:55.850396605 +0000 UTC m=+130.272759284 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : secret "metrics-daemon-secret" not found
Sep 09 21:39:51.852064 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.852050    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 21:39:51.862988 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.862967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hqx\" (UniqueName: \"kubernetes.io/projected/e7e4dd6f-2775-469a-af85-3e5392ceaf31-kube-api-access-78hqx\") pod \"network-check-target-zdcfg\" (UID: \"e7e4dd6f-2775-469a-af85-3e5392ceaf31\") " pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:51.898235 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.898204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6js7k\""
Sep 09 21:39:51.901044 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.901024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kpfp"
Sep 09 21:39:51.906223 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:51.906201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:52.202443 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:52.202409    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9kpfp"]
Sep 09 21:39:52.206708 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:39:52.206684    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c997d6_cc19_4559_8591_58462280463f.slice/crio-38ff10320bb96feddd21962d06c62aace3835c9edd6a312ea49063036d9474b2 WatchSource:0}: Error finding container 38ff10320bb96feddd21962d06c62aace3835c9edd6a312ea49063036d9474b2: Status 404 returned error can't find the container with id 38ff10320bb96feddd21962d06c62aace3835c9edd6a312ea49063036d9474b2
Sep 09 21:39:52.225868 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:52.225839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zdcfg"]
Sep 09 21:39:52.229169 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:39:52.229144    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e4dd6f_2775_469a_af85_3e5392ceaf31.slice/crio-9e180962634a3de5b36af22e78f1e0eb5fcffea0b74b465593443dca3a3b2015 WatchSource:0}: Error finding container 9e180962634a3de5b36af22e78f1e0eb5fcffea0b74b465593443dca3a3b2015: Status 404 returned error can't find the container with id 9e180962634a3de5b36af22e78f1e0eb5fcffea0b74b465593443dca3a3b2015
Sep 09 21:39:52.323448 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:52.323374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9kpfp" event={"ID":"99c997d6-cc19-4559-8591-58462280463f","Type":"ContainerStarted","Data":"38ff10320bb96feddd21962d06c62aace3835c9edd6a312ea49063036d9474b2"}
Sep 09 21:39:52.324409 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:52.324384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zdcfg" event={"ID":"e7e4dd6f-2775-469a-af85-3e5392ceaf31","Type":"ContainerStarted","Data":"9e180962634a3de5b36af22e78f1e0eb5fcffea0b74b465593443dca3a3b2015"}
Sep 09 21:39:53.934961 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:53.934924    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"]
Sep 09 21:39:53.938683 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:53.937458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"
Sep 09 21:39:53.940492 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:53.940462    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 21:39:53.941917 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:53.941649    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 21:39:53.941917 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:53.941681    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 21:39:53.941917 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:53.941650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xcmvm\""
Sep 09 21:39:53.942163 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:53.941643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 21:39:53.946969 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:53.946951    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"]
Sep 09 21:39:54.058360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:54.058317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f38d52c6-f1ba-4697-80c3-d2830f43e71b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c64f446f6-qjlsb\" (UID: \"f38d52c6-f1ba-4697-80c3-d2830f43e71b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"
Sep 09 21:39:54.058551 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:54.058371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5fv\" (UniqueName: \"kubernetes.io/projected/f38d52c6-f1ba-4697-80c3-d2830f43e71b-kube-api-access-sp5fv\") pod \"managed-serviceaccount-addon-agent-c64f446f6-qjlsb\" (UID: \"f38d52c6-f1ba-4697-80c3-d2830f43e71b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"
Sep 09 21:39:54.159792 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:54.159754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f38d52c6-f1ba-4697-80c3-d2830f43e71b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c64f446f6-qjlsb\" (UID: \"f38d52c6-f1ba-4697-80c3-d2830f43e71b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"
Sep 09 21:39:54.159990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:54.159800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5fv\" (UniqueName: \"kubernetes.io/projected/f38d52c6-f1ba-4697-80c3-d2830f43e71b-kube-api-access-sp5fv\") pod \"managed-serviceaccount-addon-agent-c64f446f6-qjlsb\" (UID: \"f38d52c6-f1ba-4697-80c3-d2830f43e71b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"
Sep 09 21:39:54.164207 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:54.164160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f38d52c6-f1ba-4697-80c3-d2830f43e71b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c64f446f6-qjlsb\" (UID: \"f38d52c6-f1ba-4697-80c3-d2830f43e71b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"
Sep 09 21:39:54.168755 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:54.168686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5fv\" (UniqueName: \"kubernetes.io/projected/f38d52c6-f1ba-4697-80c3-d2830f43e71b-kube-api-access-sp5fv\") pod \"managed-serviceaccount-addon-agent-c64f446f6-qjlsb\" (UID: \"f38d52c6-f1ba-4697-80c3-d2830f43e71b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"
Sep 09 21:39:54.262048 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:54.261952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"
Sep 09 21:39:56.818021 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:56.817937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb"]
Sep 09 21:39:56.821115 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:39:56.821086    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf38d52c6_f1ba_4697_80c3_d2830f43e71b.slice/crio-6a899249fcc573009b8fffedf5d0d0033adbc6ea3afef664e92012595e389fef WatchSource:0}: Error finding container 6a899249fcc573009b8fffedf5d0d0033adbc6ea3afef664e92012595e389fef: Status 404 returned error can't find the container with id 6a899249fcc573009b8fffedf5d0d0033adbc6ea3afef664e92012595e389fef
Sep 09 21:39:57.341475 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:57.341184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zdcfg" event={"ID":"e7e4dd6f-2775-469a-af85-3e5392ceaf31","Type":"ContainerStarted","Data":"8654b826954d3698b6ef628a68ec5c38d83f40448e2e0fc4a28e0122eede3c5c"}
Sep 09 21:39:57.341681 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:57.341555    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:39:57.343788 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:57.343756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9kpfp" event={"ID":"99c997d6-cc19-4559-8591-58462280463f","Type":"ContainerStarted","Data":"468bdc1d01e7f49b0e853e0cc00b2f3d932d8cdb99bd8138d0d61745204c3fba"}
Sep 09 21:39:57.344846 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:57.344820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb" event={"ID":"f38d52c6-f1ba-4697-80c3-d2830f43e71b","Type":"ContainerStarted","Data":"6a899249fcc573009b8fffedf5d0d0033adbc6ea3afef664e92012595e389fef"}
Sep 09 21:39:57.356936 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:57.356874    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zdcfg" podStartSLOduration=67.025931999 podStartE2EDuration="1m11.356854529s" totalImagesPullingTime="4.33092253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:57.356756876 +0000 UTC m=+71.779119573" watchObservedRunningTime="2026-09-09 21:39:57.356854529 +0000 UTC m=+71.779217229"
Sep 09 21:39:57.372953 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:57.371329    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9kpfp" podStartSLOduration=67.01399473 podStartE2EDuration="1m11.371305458s" totalImagesPullingTime="4.357310728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:57.369986597 +0000 UTC m=+71.792349288" watchObservedRunningTime="2026-09-09 21:39:57.371305458 +0000 UTC m=+71.793668160"
Sep 09 21:39:59.351056 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:59.351022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb" event={"ID":"f38d52c6-f1ba-4697-80c3-d2830f43e71b","Type":"ContainerStarted","Data":"eb991f59feac5e55fbb298a7417e5cb97a7fadaee8de977f02d01e09c897f1b7"}
Sep 09 21:39:59.365599 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:39:59.365493    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb" podStartSLOduration=4.098262602 podStartE2EDuration="6.365472935s" totalImagesPullingTime="2.267210333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:59.365060234 +0000 UTC m=+73.787422946" watchObservedRunningTime="2026-09-09 21:39:59.365472935 +0000 UTC m=+73.787835636"
Sep 09 21:40:23.360935 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:40:23.360814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:40:23.361376 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:40:23.360966    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:40:23.361376 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:40:23.361032    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:41:27.361016987 +0000 UTC m=+161.783379669 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:40:28.349794 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:40:28.349754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zdcfg"
Sep 09 21:40:55.877574 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:40:55.877529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:40:55.878137 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:40:55.877678    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 21:40:55.878137 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:40:55.877735    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs podName:048fbca3-d90e-4740-a1d0-1b7f77453e04 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:57.877719506 +0000 UTC m=+252.300082184 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs") pod "network-metrics-daemon-wvv59" (UID: "048fbca3-d90e-4740-a1d0-1b7f77453e04") : secret "metrics-daemon-secret" not found
Sep 09 21:41:22.514224 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:41:22.514170    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-v99wb" podUID="d834a708-778c-44a5-ad5a-ecdcb56aa53f"
Sep 09 21:41:22.558229 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:41:22.558205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v99wb"
Sep 09 21:41:23.105594 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:41:23.105560    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wvv59" podUID="048fbca3-d90e-4740-a1d0-1b7f77453e04"
Sep 09 21:41:27.386552 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:41:27.386499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:41:27.387116 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:41:27.386674    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 21:41:27.387116 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:41:27.386736    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls podName:d834a708-778c-44a5-ad5a-ecdcb56aa53f nodeName:}" failed. No retries permitted until 2026-09-09 21:43:29.386718783 +0000 UTC m=+283.809081481 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls") pod "dns-default-v99wb" (UID: "d834a708-778c-44a5-ad5a-ecdcb56aa53f") : secret "dns-default-metrics-tls" not found
Sep 09 21:41:35.086210 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:41:35.086157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:41:59.651902 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:41:59.651865    2579 generic.go:356] "Generic (PLEG): container finished" podID="f38d52c6-f1ba-4697-80c3-d2830f43e71b" containerID="eb991f59feac5e55fbb298a7417e5cb97a7fadaee8de977f02d01e09c897f1b7" exitCode=255
Sep 09 21:41:59.652430 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:41:59.651945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb" event={"ID":"f38d52c6-f1ba-4697-80c3-d2830f43e71b","Type":"ContainerDied","Data":"eb991f59feac5e55fbb298a7417e5cb97a7fadaee8de977f02d01e09c897f1b7"}
Sep 09 21:41:59.658038 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:41:59.658021    2579 scope.go:117] "RemoveContainer" containerID="eb991f59feac5e55fbb298a7417e5cb97a7fadaee8de977f02d01e09c897f1b7"
Sep 09 21:42:00.656468 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:00.656434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c64f446f6-qjlsb" event={"ID":"f38d52c6-f1ba-4697-80c3-d2830f43e71b","Type":"ContainerStarted","Data":"c41d02ce6cad4ed1b3d045d07f6a149eddc6bbd24e202943ea0e2b1cd8d4a65f"}
Sep 09 21:42:35.080577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.080541    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bc57fd5bc-rfskd"]
Sep 09 21:42:35.082461 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.082445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.084578 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.084554    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7c55bbf95b-wdt7z"]
Sep 09 21:42:35.084770 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.084750    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 21:42:35.084843 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.084784    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 21:42:35.084843 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.084827    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xvgqt\""
Sep 09 21:42:35.085154 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.085133    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 21:42:35.086558 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.086536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.089074 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.089055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 21:42:35.089220 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.089204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xlrrp\""
Sep 09 21:42:35.089461 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.089444    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 21:42:35.089563 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.089545    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 21:42:35.089650 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.089592    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 21:42:35.091761 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.091741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 21:42:35.091871 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.091806    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 21:42:35.092950 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.092929    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 21:42:35.097683 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.097663    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bc57fd5bc-rfskd"]
Sep 09 21:42:35.104262 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.104242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c55bbf95b-wdt7z"]
Sep 09 21:42:35.138020 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.137991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-image-registry-private-configuration\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.138020 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-certificates\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.138177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmsc\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-kube-api-access-qpmsc\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.138177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-installation-pull-secrets\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.138177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.138177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-ca-trust-extracted\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.138341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-default-certificate\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.138341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-bound-sa-token\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.138341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.138341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-stats-auth\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.138341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-trusted-ca\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.138341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6h4\" (UniqueName: \"kubernetes.io/projected/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-kube-api-access-jw6h4\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.138341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.138308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.176241 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.176216    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-mtj6v"]
Sep 09 21:42:35.178315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.178296    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6z657"]
Sep 09 21:42:35.178459 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.178439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-mtj6v"
Sep 09 21:42:35.179980 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.179966    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"]
Sep 09 21:42:35.180097 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.180082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.180968 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.180952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dj4zk\""
Sep 09 21:42:35.181743 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.181726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.183703 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.183680    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 21:42:35.183858 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.183708    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 21:42:35.184015 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.183999    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-j4pm7\""
Sep 09 21:42:35.184373 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.184350    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:42:35.184469 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.184361    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 21:42:35.184469 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.184394    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 21:42:35.184594 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.184576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 21:42:35.184671 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.184593    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 21:42:35.184841 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.184724    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 21:42:35.184898 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.184871    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-45vm6\""
Sep 09 21:42:35.190635 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.190618    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 21:42:35.194082 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.194057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"]
Sep 09 21:42:35.194888 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.194865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6z657"]
Sep 09 21:42:35.195677 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.195656    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-mtj6v"]
Sep 09 21:42:35.238647 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.238647 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-snapshots\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.238883 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-ca-trust-extracted\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.238883 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.238883 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-bound-sa-token\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.238883 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-tmp\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.238883 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.238883 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-serving-cert\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.238883 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.238781    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:35.738756102 +0000 UTC m=+230.161118796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : configmap references non-existent config key: service-ca.crt
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.238912    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.238949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/598d76a6-65f4-48d0-8deb-96c1db8c084a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.238964    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:35.738951346 +0000 UTC m=+230.161314024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : secret "router-metrics-certs-default" not found
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ghh\" (UniqueName: \"kubernetes.io/projected/40c164a2-e138-438a-9b87-08dff7fe028d-kube-api-access-r8ghh\") pod \"network-check-source-685456655f-mtj6v\" (UID: \"40c164a2-e138-438a-9b87-08dff7fe028d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-mtj6v"
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-default-certificate\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-stats-auth\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-trusted-ca\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-ca-trust-extracted\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.239144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6h4\" (UniqueName: \"kubernetes.io/projected/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-kube-api-access-jw6h4\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/598d76a6-65f4-48d0-8deb-96c1db8c084a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s698j\" (UniqueName: \"kubernetes.io/projected/598d76a6-65f4-48d0-8deb-96c1db8c084a-kube-api-access-s698j\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-installation-pull-secrets\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.239288    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.239303    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bc57fd5bc-rfskd: secret "image-registry-tls" not found
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-certificates\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.239522 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.239407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls podName:11278b72-85bd-4ca1-9ce0-904d0aa6ad00 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:35.739384768 +0000 UTC m=+230.161747466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls") pod "image-registry-7bc57fd5bc-rfskd" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00") : secret "image-registry-tls" not found
Sep 09 21:42:35.239990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8zl\" (UniqueName: \"kubernetes.io/projected/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-kube-api-access-ll8zl\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.239990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-image-registry-private-configuration\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.239990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmsc\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-kube-api-access-qpmsc\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.240109 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.239990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-certificates\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.240250 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.240227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-trusted-ca\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.241766 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.241737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-installation-pull-secrets\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.241863 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.241795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-image-registry-private-configuration\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.242078 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.242054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-stats-auth\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.242195 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.242180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-default-certificate\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.246878 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.246828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-bound-sa-token\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.247235 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.247208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6h4\" (UniqueName: \"kubernetes.io/projected/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-kube-api-access-jw6h4\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.248023 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.248001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmsc\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-kube-api-access-qpmsc\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.340401 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-tmp\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.340401 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.340401 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-serving-cert\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.340401 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/598d76a6-65f4-48d0-8deb-96c1db8c084a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.340793 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ghh\" (UniqueName: \"kubernetes.io/projected/40c164a2-e138-438a-9b87-08dff7fe028d-kube-api-access-r8ghh\") pod \"network-check-source-685456655f-mtj6v\" (UID: \"40c164a2-e138-438a-9b87-08dff7fe028d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-mtj6v"
Sep 09 21:42:35.340793 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/598d76a6-65f4-48d0-8deb-96c1db8c084a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.340793 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s698j\" (UniqueName: \"kubernetes.io/projected/598d76a6-65f4-48d0-8deb-96c1db8c084a-kube-api-access-s698j\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.340793 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.340793 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8zl\" (UniqueName: \"kubernetes.io/projected/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-kube-api-access-ll8zl\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.340793 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-tmp\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.340793 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.340793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-snapshots\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.341314 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.341292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.341388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.341314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/598d76a6-65f4-48d0-8deb-96c1db8c084a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.341388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.341369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.341455 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.341410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-snapshots\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.342895 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.342873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/598d76a6-65f4-48d0-8deb-96c1db8c084a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.342999 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.342978    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-serving-cert\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.348175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.348153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s698j\" (UniqueName: \"kubernetes.io/projected/598d76a6-65f4-48d0-8deb-96c1db8c084a-kube-api-access-s698j\") pod \"kube-storage-version-migrator-operator-8d7dd4675-25d5s\" (UID: \"598d76a6-65f4-48d0-8deb-96c1db8c084a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.348275 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.348255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8zl\" (UniqueName: \"kubernetes.io/projected/c0aa425c-e7ed-4feb-9610-55b9cf1ee45e-kube-api-access-ll8zl\") pod \"insights-operator-76bcd6ccf5-6z657\" (UID: \"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.348324 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.348279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ghh\" (UniqueName: \"kubernetes.io/projected/40c164a2-e138-438a-9b87-08dff7fe028d-kube-api-access-r8ghh\") pod \"network-check-source-685456655f-mtj6v\" (UID: \"40c164a2-e138-438a-9b87-08dff7fe028d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-mtj6v"
Sep 09 21:42:35.488135 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.488096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-mtj6v"
Sep 09 21:42:35.494931 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.494914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-6z657"
Sep 09 21:42:35.501769 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.501750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"
Sep 09 21:42:35.745325 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.745296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:35.745509 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.745347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.745509 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.745369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:35.745509 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.745459    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 21:42:35.745509 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.745462    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:42:35.745509 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.745477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:36.74545926 +0000 UTC m=+231.167821938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : configmap references non-existent config key: service-ca.crt
Sep 09 21:42:35.745509 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.745485    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bc57fd5bc-rfskd: secret "image-registry-tls" not found
Sep 09 21:42:35.745509 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.745497    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:36.745490059 +0000 UTC m=+231.167852737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : secret "router-metrics-certs-default" not found
Sep 09 21:42:35.745815 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:35.745531    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls podName:11278b72-85bd-4ca1-9ce0-904d0aa6ad00 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:36.745514734 +0000 UTC m=+231.167877425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls") pod "image-registry-7bc57fd5bc-rfskd" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00") : secret "image-registry-tls" not found
Sep 09 21:42:35.756089 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.756064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-mtj6v"]
Sep 09 21:42:35.758662 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:42:35.758640    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c164a2_e138_438a_9b87_08dff7fe028d.slice/crio-3b7a535d0719589571a0fd791ce70aeaf92cfd76b3c22978b46e1727178b8c56 WatchSource:0}: Error finding container 3b7a535d0719589571a0fd791ce70aeaf92cfd76b3c22978b46e1727178b8c56: Status 404 returned error can't find the container with id 3b7a535d0719589571a0fd791ce70aeaf92cfd76b3c22978b46e1727178b8c56
Sep 09 21:42:35.779951 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.779931    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s"]
Sep 09 21:42:35.782834 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:42:35.782810    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598d76a6_65f4_48d0_8deb_96c1db8c084a.slice/crio-a759aff7377a6357f7ea8995d75fb189b2ef5df5ffeacd8c00ef3a31b03dffa6 WatchSource:0}: Error finding container a759aff7377a6357f7ea8995d75fb189b2ef5df5ffeacd8c00ef3a31b03dffa6: Status 404 returned error can't find the container with id a759aff7377a6357f7ea8995d75fb189b2ef5df5ffeacd8c00ef3a31b03dffa6
Sep 09 21:42:35.783329 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:35.783308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6z657"]
Sep 09 21:42:35.787098 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:42:35.787076    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0aa425c_e7ed_4feb_9610_55b9cf1ee45e.slice/crio-9aa8ee0e882af09d38ba151fe92c546409d61e4b5fbf247d350f964a4acaef49 WatchSource:0}: Error finding container 9aa8ee0e882af09d38ba151fe92c546409d61e4b5fbf247d350f964a4acaef49: Status 404 returned error can't find the container with id 9aa8ee0e882af09d38ba151fe92c546409d61e4b5fbf247d350f964a4acaef49
Sep 09 21:42:36.746365 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:36.746112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-mtj6v" event={"ID":"40c164a2-e138-438a-9b87-08dff7fe028d","Type":"ContainerStarted","Data":"a72e42067161669c97ceea153581ee5b602e7c1359295f4fda35187bc01dce27"}
Sep 09 21:42:36.746365 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:36.746188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-mtj6v" event={"ID":"40c164a2-e138-438a-9b87-08dff7fe028d","Type":"ContainerStarted","Data":"3b7a535d0719589571a0fd791ce70aeaf92cfd76b3c22978b46e1727178b8c56"}
Sep 09 21:42:36.747921 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:36.747863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s" event={"ID":"598d76a6-65f4-48d0-8deb-96c1db8c084a","Type":"ContainerStarted","Data":"a759aff7377a6357f7ea8995d75fb189b2ef5df5ffeacd8c00ef3a31b03dffa6"}
Sep 09 21:42:36.749184 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:36.749157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6z657" event={"ID":"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e","Type":"ContainerStarted","Data":"9aa8ee0e882af09d38ba151fe92c546409d61e4b5fbf247d350f964a4acaef49"}
Sep 09 21:42:36.754324 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:36.754294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:36.754435 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:36.754334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:36.754435 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:36.754423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:36.754550 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:36.754453    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 21:42:36.754550 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:36.754511    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:38.75449261 +0000 UTC m=+233.176855306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : secret "router-metrics-certs-default" not found
Sep 09 21:42:36.754550 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:36.754544    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:42:36.754728 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:36.754556    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bc57fd5bc-rfskd: secret "image-registry-tls" not found
Sep 09 21:42:36.754728 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:36.754581    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:38.754568381 +0000 UTC m=+233.176931073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : configmap references non-existent config key: service-ca.crt
Sep 09 21:42:36.754728 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:36.754596    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls podName:11278b72-85bd-4ca1-9ce0-904d0aa6ad00 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:38.754588788 +0000 UTC m=+233.176951466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls") pod "image-registry-7bc57fd5bc-rfskd" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00") : secret "image-registry-tls" not found
Sep 09 21:42:36.761511 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:36.761453    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-mtj6v" podStartSLOduration=1.7614321130000001 podStartE2EDuration="1.761432113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:36.759974184 +0000 UTC m=+231.182336888" watchObservedRunningTime="2026-09-09 21:42:36.761432113 +0000 UTC m=+231.183794815"
Sep 09 21:42:38.759787 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:38.759746    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s" event={"ID":"598d76a6-65f4-48d0-8deb-96c1db8c084a","Type":"ContainerStarted","Data":"120fcc22fefdd0b2d14e574a224aded881a9ad59a36ff675942efff760f3b43d"}
Sep 09 21:42:38.761335 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:38.761302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6z657" event={"ID":"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e","Type":"ContainerStarted","Data":"88008589dee5f3686702b763f15b2f80bafbb732bc7a09f695158a1f62a8c788"}
Sep 09 21:42:38.774364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:38.774332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:38.774482 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:38.774420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:38.774482 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:38.774449    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:42:38.774482 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:38.774469    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bc57fd5bc-rfskd: secret "image-registry-tls" not found
Sep 09 21:42:38.774664 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:38.774456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:38.774664 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:38.774525    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls podName:11278b72-85bd-4ca1-9ce0-904d0aa6ad00 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:42.774506859 +0000 UTC m=+237.196869537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls") pod "image-registry-7bc57fd5bc-rfskd" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00") : secret "image-registry-tls" not found
Sep 09 21:42:38.774664 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:38.774537    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 21:42:38.774664 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:38.774575    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:42.774560392 +0000 UTC m=+237.196923094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : configmap references non-existent config key: service-ca.crt
Sep 09 21:42:38.774664 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:38.774611    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:42.774593848 +0000 UTC m=+237.196956527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : secret "router-metrics-certs-default" not found
Sep 09 21:42:38.777654 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:38.777616    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s" podStartSLOduration=1.228703932 podStartE2EDuration="3.777587895s" totalImagesPullingTime="2.548883963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:38.776861552 +0000 UTC m=+233.199224254" watchObservedRunningTime="2026-09-09 21:42:38.777587895 +0000 UTC m=+233.199950633"
Sep 09 21:42:38.797807 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:38.797745    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-6z657" podStartSLOduration=1.257659876 podStartE2EDuration="3.797730491s" totalImagesPullingTime="2.540070615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:38.796778742 +0000 UTC m=+233.219141445" watchObservedRunningTime="2026-09-09 21:42:38.797730491 +0000 UTC m=+233.220093190"
Sep 09 21:42:39.502255 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.502213    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vfcbw"]
Sep 09 21:42:39.504704 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.504634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.506858 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.506839    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 21:42:39.506858 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.506851    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 21:42:39.507039 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.506874    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jhfz7\""
Sep 09 21:42:39.517013 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.516989    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vfcbw"]
Sep 09 21:42:39.581428 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.581395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.581616 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.581442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5nsc\" (UniqueName: \"kubernetes.io/projected/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-api-access-c5nsc\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.581616 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.581508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-crio-socket\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.581616 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.581526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-data-volume\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.581616 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.581545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.682231 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.682198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5nsc\" (UniqueName: \"kubernetes.io/projected/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-api-access-c5nsc\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.682422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.682254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-crio-socket\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.682422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.682276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-data-volume\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.682422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.682293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.682422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.682359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.682422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.682366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-crio-socket\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.682689 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:39.682458    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:42:39.682689 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:39.682481    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:39.682689 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:39.682520    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm podName:92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:40.182500304 +0000 UTC m=+234.604862989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfcbw" (UID: "92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3") : configmap "kube-rbac-proxy" not found
Sep 09 21:42:39.682689 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:39.682547    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls podName:92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:40.182529606 +0000 UTC m=+234.604892289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfcbw" (UID: "92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:39.682689 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.682638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-data-volume\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:39.690213 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:39.690190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5nsc\" (UniqueName: \"kubernetes.io/projected/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-api-access-c5nsc\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:40.186961 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.186922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:40.187377 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.186982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:40.187377 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:40.187040    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:42:40.187377 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:40.187098    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:40.187377 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:40.187107    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm podName:92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:41.187091793 +0000 UTC m=+235.609454471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfcbw" (UID: "92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3") : configmap "kube-rbac-proxy" not found
Sep 09 21:42:40.187377 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:40.187139    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls podName:92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:41.187128368 +0000 UTC m=+235.609491051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfcbw" (UID: "92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:40.328901 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.328869    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6"]
Sep 09 21:42:40.331025 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.331009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6"
Sep 09 21:42:40.334127 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.334106    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 21:42:40.334250 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.334149    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kmfbx\""
Sep 09 21:42:40.335495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.335274    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 21:42:40.341874 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.341851    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6"]
Sep 09 21:42:40.388139 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.388112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzjj\" (UniqueName: \"kubernetes.io/projected/9141d428-585e-4ca7-8aa9-7ccc8250b425-kube-api-access-2kzjj\") pod \"migrator-fdbb8fdff-cl7b6\" (UID: \"9141d428-585e-4ca7-8aa9-7ccc8250b425\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6"
Sep 09 21:42:40.488930 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.488841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzjj\" (UniqueName: \"kubernetes.io/projected/9141d428-585e-4ca7-8aa9-7ccc8250b425-kube-api-access-2kzjj\") pod \"migrator-fdbb8fdff-cl7b6\" (UID: \"9141d428-585e-4ca7-8aa9-7ccc8250b425\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6"
Sep 09 21:42:40.496803 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.496775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzjj\" (UniqueName: \"kubernetes.io/projected/9141d428-585e-4ca7-8aa9-7ccc8250b425-kube-api-access-2kzjj\") pod \"migrator-fdbb8fdff-cl7b6\" (UID: \"9141d428-585e-4ca7-8aa9-7ccc8250b425\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6"
Sep 09 21:42:40.642435 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.642402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6"
Sep 09 21:42:40.904525 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:40.904494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6"]
Sep 09 21:42:40.907591 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:42:40.907562    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9141d428_585e_4ca7_8aa9_7ccc8250b425.slice/crio-acadfc441c0850e9dabb97bef0707d043ecf2c1e54ddcb50cc4c0cc8b627ae13 WatchSource:0}: Error finding container acadfc441c0850e9dabb97bef0707d043ecf2c1e54ddcb50cc4c0cc8b627ae13: Status 404 returned error can't find the container with id acadfc441c0850e9dabb97bef0707d043ecf2c1e54ddcb50cc4c0cc8b627ae13
Sep 09 21:42:41.194491 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:41.194391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:41.194491 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:41.194456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:41.194920 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:41.194522    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 21:42:41.194920 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:41.194541    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:41.194920 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:41.194595    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm podName:92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:43.194575574 +0000 UTC m=+237.616938272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfcbw" (UID: "92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3") : configmap "kube-rbac-proxy" not found
Sep 09 21:42:41.194920 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:41.194632    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls podName:92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:43.194622936 +0000 UTC m=+237.616985628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfcbw" (UID: "92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:41.681571 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:41.681538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vg5zh_f86d9c73-bc49-4713-96dd-4b96e1cf9156/dns-node-resolver/0.log"
Sep 09 21:42:41.772418 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:41.772381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6" event={"ID":"9141d428-585e-4ca7-8aa9-7ccc8250b425","Type":"ContainerStarted","Data":"acadfc441c0850e9dabb97bef0707d043ecf2c1e54ddcb50cc4c0cc8b627ae13"}
Sep 09 21:42:42.481394 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:42.481322    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d8dsb_9bbe1e6f-07bc-4d52-b391-79178b7b6be9/node-ca/0.log"
Sep 09 21:42:42.777538 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:42.777450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6" event={"ID":"9141d428-585e-4ca7-8aa9-7ccc8250b425","Type":"ContainerStarted","Data":"e8d82c018811f35135062f1d80603d005db3165ebe062c4a8f01d6c66d8bb14b"}
Sep 09 21:42:42.777538 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:42.777511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6" event={"ID":"9141d428-585e-4ca7-8aa9-7ccc8250b425","Type":"ContainerStarted","Data":"80c183534e92cd4d507eebe77250b3320d8fa3098ac0b002e8e1894ba00a7572"}
Sep 09 21:42:42.794346 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:42.794293    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-cl7b6" podStartSLOduration=1.5479937449999999 podStartE2EDuration="2.79427797s" totalImagesPullingTime="1.246284225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:42.793198446 +0000 UTC m=+237.215561149" watchObservedRunningTime="2026-09-09 21:42:42.79427797 +0000 UTC m=+237.216640670"
Sep 09 21:42:42.807286 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:42.807254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:42.807349 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:42.807310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:42.807349 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:42.807332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:42.807419 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:42.807399    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 21:42:42.807419 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:42.807417    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bc57fd5bc-rfskd: secret "image-registry-tls" not found
Sep 09 21:42:42.807489 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:42.807434    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 21:42:42.807489 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:42.807469    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls podName:11278b72-85bd-4ca1-9ce0-904d0aa6ad00 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:50.807453212 +0000 UTC m=+245.229815919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls") pod "image-registry-7bc57fd5bc-rfskd" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00") : secret "image-registry-tls" not found
Sep 09 21:42:42.807489 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:42.807483    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:50.807477368 +0000 UTC m=+245.229840046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : secret "router-metrics-certs-default" not found
Sep 09 21:42:42.807628 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:42.807491    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle podName:5613aec3-899d-4cbf-a406-7e1d4b0f5bd7 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:50.807486856 +0000 UTC m=+245.229849533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle") pod "router-default-7c55bbf95b-wdt7z" (UID: "5613aec3-899d-4cbf-a406-7e1d4b0f5bd7") : configmap references non-existent config key: service-ca.crt
Sep 09 21:42:43.211269 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:43.211235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:43.211459 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:43.211342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:43.211459 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:43.211380    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:43.211459 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:43.211441    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls podName:92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:47.211427242 +0000 UTC m=+241.633789925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfcbw" (UID: "92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:43.211918 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:43.211897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:43.281539 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:43.281505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mfrlp_667ff566-22cb-4cdc-beb3-6bab86102ec5/serve-healthcheck-canary/0.log"
Sep 09 21:42:47.244762 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:47.244723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:47.245137 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:47.244891    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:47.245137 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:42:47.244961    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls podName:92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3 nodeName:}" failed. No retries permitted until 2026-09-09 21:42:55.244944922 +0000 UTC m=+249.667307600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfcbw" (UID: "92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 21:42:50.873050 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:50.873010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:50.873444 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:50.873066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:50.873444 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:50.873090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:50.873694 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:50.873673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-service-ca-bundle\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:50.875466 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:50.875444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5613aec3-899d-4cbf-a406-7e1d4b0f5bd7-metrics-certs\") pod \"router-default-7c55bbf95b-wdt7z\" (UID: \"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7\") " pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:50.875578 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:50.875561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"image-registry-7bc57fd5bc-rfskd\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") " pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:50.996662 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:50.996635    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xvgqt\""
Sep 09 21:42:51.001581 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.001559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xlrrp\""
Sep 09 21:42:51.004287 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.004268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:51.009837 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.009820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:51.275708 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.275593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bc57fd5bc-rfskd"]
Sep 09 21:42:51.279071 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:42:51.279047    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11278b72_85bd_4ca1_9ce0_904d0aa6ad00.slice/crio-67ee295a1146109a07a6609c89a547fb44f0efe7339cd441afd8c1c04dca004e WatchSource:0}: Error finding container 67ee295a1146109a07a6609c89a547fb44f0efe7339cd441afd8c1c04dca004e: Status 404 returned error can't find the container with id 67ee295a1146109a07a6609c89a547fb44f0efe7339cd441afd8c1c04dca004e
Sep 09 21:42:51.281495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.281469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c55bbf95b-wdt7z"]
Sep 09 21:42:51.283652 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:42:51.283631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5613aec3_899d_4cbf_a406_7e1d4b0f5bd7.slice/crio-01c3eaa5cc80c1e2271b76b06004d569004e5a2e920a056ac0d2912b41511c25 WatchSource:0}: Error finding container 01c3eaa5cc80c1e2271b76b06004d569004e5a2e920a056ac0d2912b41511c25: Status 404 returned error can't find the container with id 01c3eaa5cc80c1e2271b76b06004d569004e5a2e920a056ac0d2912b41511c25
Sep 09 21:42:51.809339 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.809306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z" event={"ID":"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7","Type":"ContainerStarted","Data":"e83372d381816bf67a3f790b2c973564e78455cb8757a1b0b571c4f2fa1887a0"}
Sep 09 21:42:51.809552 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.809359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z" event={"ID":"5613aec3-899d-4cbf-a406-7e1d4b0f5bd7","Type":"ContainerStarted","Data":"01c3eaa5cc80c1e2271b76b06004d569004e5a2e920a056ac0d2912b41511c25"}
Sep 09 21:42:51.810884 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.810858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd" event={"ID":"11278b72-85bd-4ca1-9ce0-904d0aa6ad00","Type":"ContainerStarted","Data":"2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be"}
Sep 09 21:42:51.810976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.810900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd" event={"ID":"11278b72-85bd-4ca1-9ce0-904d0aa6ad00","Type":"ContainerStarted","Data":"67ee295a1146109a07a6609c89a547fb44f0efe7339cd441afd8c1c04dca004e"}
Sep 09 21:42:51.810976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.810922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:42:51.827088 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.827039    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z" podStartSLOduration=16.827026488 podStartE2EDuration="16.827026488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:51.826377266 +0000 UTC m=+246.248739967" watchObservedRunningTime="2026-09-09 21:42:51.827026488 +0000 UTC m=+246.249389228"
Sep 09 21:42:51.842735 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:51.842695    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd" podStartSLOduration=16.842679991 podStartE2EDuration="16.842679991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:51.842090551 +0000 UTC m=+246.264453290" watchObservedRunningTime="2026-09-09 21:42:51.842679991 +0000 UTC m=+246.265042691"
Sep 09 21:42:52.010480 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:52.010432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:52.013274 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:52.013247    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:52.814347 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:52.814312    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:52.815540 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:52.815518    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7c55bbf95b-wdt7z"
Sep 09 21:42:55.311001 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:55.310963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:55.313207 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:55.313187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfcbw\" (UID: \"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3\") " pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:55.417083 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:55.417048    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jhfz7\""
Sep 09 21:42:55.424859 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:55.424838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vfcbw"
Sep 09 21:42:55.692637 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:55.692537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vfcbw"]
Sep 09 21:42:55.695349 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:42:55.695322    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e50cc2_321e_4ddc_bf0a_c0cbcb7df1e3.slice/crio-d6e16fa44064fb39e944f55e3d55d180acea2248eb7c6f4873601ac94082a79e WatchSource:0}: Error finding container d6e16fa44064fb39e944f55e3d55d180acea2248eb7c6f4873601ac94082a79e: Status 404 returned error can't find the container with id d6e16fa44064fb39e944f55e3d55d180acea2248eb7c6f4873601ac94082a79e
Sep 09 21:42:55.825057 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:55.825025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfcbw" event={"ID":"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3","Type":"ContainerStarted","Data":"d1264a39b98959939e6524b77ace89ab45e60f7cdca894546b061197aa89b668"}
Sep 09 21:42:55.825203 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:55.825085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfcbw" event={"ID":"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3","Type":"ContainerStarted","Data":"d6e16fa44064fb39e944f55e3d55d180acea2248eb7c6f4873601ac94082a79e"}
Sep 09 21:42:56.830861 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:56.830781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfcbw" event={"ID":"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3","Type":"ContainerStarted","Data":"5333f2ff46dd5d65676dab3b73f0425d9b03659ed8660fda03eafda7b99be4af"}
Sep 09 21:42:57.929418 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:57.929383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:42:57.931844 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:57.931820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fbca3-d90e-4740-a1d0-1b7f77453e04-metrics-certs\") pod \"network-metrics-daemon-wvv59\" (UID: \"048fbca3-d90e-4740-a1d0-1b7f77453e04\") " pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:42:58.189831 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:58.189743    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-znm4z\""
Sep 09 21:42:58.197669 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:58.197646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wvv59"
Sep 09 21:42:58.463332 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:58.463146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wvv59"]
Sep 09 21:42:58.465661 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:42:58.465634    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048fbca3_d90e_4740_a1d0_1b7f77453e04.slice/crio-c6acbcfb415c6cb458d290b2505128f5070b3e57c121ae78096f934e065d530d WatchSource:0}: Error finding container c6acbcfb415c6cb458d290b2505128f5070b3e57c121ae78096f934e065d530d: Status 404 returned error can't find the container with id c6acbcfb415c6cb458d290b2505128f5070b3e57c121ae78096f934e065d530d
Sep 09 21:42:58.838674 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:58.838565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wvv59" event={"ID":"048fbca3-d90e-4740-a1d0-1b7f77453e04","Type":"ContainerStarted","Data":"c6acbcfb415c6cb458d290b2505128f5070b3e57c121ae78096f934e065d530d"}
Sep 09 21:42:58.840553 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:58.840526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfcbw" event={"ID":"92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3","Type":"ContainerStarted","Data":"11692377d5a90d07d089810d67cabe389ed1cce13e11adcdbfe8692b26ae8ec7"}
Sep 09 21:42:58.863824 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:58.863775    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vfcbw" podStartSLOduration=17.726173037 podStartE2EDuration="19.863759282s" totalImagesPullingTime="2.137586245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:42:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:58.862220741 +0000 UTC m=+253.284583440" watchObservedRunningTime="2026-09-09 21:42:58.863759282 +0000 UTC m=+253.286121981"
Sep 09 21:42:59.846763 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:59.846659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wvv59" event={"ID":"048fbca3-d90e-4740-a1d0-1b7f77453e04","Type":"ContainerStarted","Data":"2c83e8700fe79fb945318c90d485240b30351af6bdb2144b4232c3726e3b82c9"}
Sep 09 21:42:59.846763 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:59.846723    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wvv59" event={"ID":"048fbca3-d90e-4740-a1d0-1b7f77453e04","Type":"ContainerStarted","Data":"2d0a0119b3c031d33d747e268dbc70363d63dae30cb585786c4650fa851c5651"}
Sep 09 21:42:59.861902 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:42:59.861849    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wvv59" podStartSLOduration=252.929685904 podStartE2EDuration="4m13.861832127s" totalImagesPullingTime="932.146223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:59.860940356 +0000 UTC m=+254.283303056" watchObservedRunningTime="2026-09-09 21:42:59.861832127 +0000 UTC m=+254.284194827"
Sep 09 21:43:00.446686 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:00.446652    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bc57fd5bc-rfskd"]
Sep 09 21:43:03.392528 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.392494    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d69696db7-slb5n"]
Sep 09 21:43:03.394778 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.394762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.396853 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.396831    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 21:43:03.397139 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.397121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 21:43:03.397907 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.397891    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 21:43:03.397972 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.397924    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 21:43:03.398024 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.397894    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 21:43:03.398314 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.398300    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 21:43:03.398428 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.398411    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hzl8x\""
Sep 09 21:43:03.398485 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.398470    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 21:43:03.405983 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.405962    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d69696db7-slb5n"]
Sep 09 21:43:03.572078 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.572047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-serving-cert\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.572078 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.572083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-oauth-serving-cert\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.572315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.572201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2p65\" (UniqueName: \"kubernetes.io/projected/100f3947-ee9c-42c4-98a8-a6d578af512d-kube-api-access-b2p65\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.572315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.572241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-oauth-config\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.572315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.572275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-service-ca\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.572315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.572310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-console-config\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.672969 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.672858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-oauth-serving-cert\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.672969 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.672946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2p65\" (UniqueName: \"kubernetes.io/projected/100f3947-ee9c-42c4-98a8-a6d578af512d-kube-api-access-b2p65\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.672969 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.672966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-oauth-config\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.673251 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.672996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-service-ca\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.673251 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.673017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-console-config\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.673251 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.673171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-serving-cert\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.673761 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.673728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-console-config\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.673906 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.673735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-oauth-serving-cert\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.673906 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.673777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-service-ca\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.675479 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.675454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-oauth-config\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.675568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.675500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-serving-cert\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.680283 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.680262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2p65\" (UniqueName: \"kubernetes.io/projected/100f3947-ee9c-42c4-98a8-a6d578af512d-kube-api-access-b2p65\") pod \"console-5d69696db7-slb5n\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") " pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.703747 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.703710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:03.970031 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:03.970007    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d69696db7-slb5n"]
Sep 09 21:43:03.972770 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:03.972746    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100f3947_ee9c_42c4_98a8_a6d578af512d.slice/crio-0a6d1587e3707273d07509fde058b536a3936e4f072c7786f9bbcd60cf948ff1 WatchSource:0}: Error finding container 0a6d1587e3707273d07509fde058b536a3936e4f072c7786f9bbcd60cf948ff1: Status 404 returned error can't find the container with id 0a6d1587e3707273d07509fde058b536a3936e4f072c7786f9bbcd60cf948ff1
Sep 09 21:43:04.865797 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:04.865758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d69696db7-slb5n" event={"ID":"100f3947-ee9c-42c4-98a8-a6d578af512d","Type":"ContainerStarted","Data":"0a6d1587e3707273d07509fde058b536a3936e4f072c7786f9bbcd60cf948ff1"}
Sep 09 21:43:06.874118 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:06.874088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d69696db7-slb5n" event={"ID":"100f3947-ee9c-42c4-98a8-a6d578af512d","Type":"ContainerStarted","Data":"9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4"}
Sep 09 21:43:06.889440 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:06.889387    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d69696db7-slb5n" podStartSLOduration=1.112289188 podStartE2EDuration="3.889371532s" totalImagesPullingTime="2.777082344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:06.889313874 +0000 UTC m=+261.311676574" watchObservedRunningTime="2026-09-09 21:43:06.889371532 +0000 UTC m=+261.311734231"
Sep 09 21:43:10.454252 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:10.454163    2579 patch_prober.go:28] interesting pod/image-registry-7bc57fd5bc-rfskd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 21:43:10.454252 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:10.454228    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd" podUID="11278b72-85bd-4ca1-9ce0-904d0aa6ad00" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:43:13.705032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:13.704987    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:13.705427 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:13.705057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:13.706317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:13.706298    2579 patch_prober.go:28] interesting pod/console-5d69696db7-slb5n container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body=
Sep 09 21:43:13.706367 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:13.706341    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d69696db7-slb5n" podUID="100f3947-ee9c-42c4-98a8-a6d578af512d" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused"
Sep 09 21:43:20.451206 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:20.451169    2579 patch_prober.go:28] interesting pod/image-registry-7bc57fd5bc-rfskd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 21:43:20.451623 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:20.451225    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd" podUID="11278b72-85bd-4ca1-9ce0-904d0aa6ad00" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:43:23.705075 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:23.705037    2579 patch_prober.go:28] interesting pod/console-5d69696db7-slb5n container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body=
Sep 09 21:43:23.705492 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:23.705100    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d69696db7-slb5n" podUID="100f3947-ee9c-42c4-98a8-a6d578af512d" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused"
Sep 09 21:43:25.466316 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.466261    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd" podUID="11278b72-85bd-4ca1-9ce0-904d0aa6ad00" containerName="registry" containerID="cri-o://2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be" gracePeriod=30
Sep 09 21:43:25.559231 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:25.559189    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-v99wb" podUID="d834a708-778c-44a5-ad5a-ecdcb56aa53f"
Sep 09 21:43:25.705447 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.705422    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:43:25.738364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.738280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") pod \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") "
Sep 09 21:43:25.738364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.738337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-installation-pull-secrets\") pod \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") "
Sep 09 21:43:25.738542 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.738370    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-image-registry-private-configuration\") pod \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") "
Sep 09 21:43:25.738542 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.738405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-trusted-ca\") pod \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") "
Sep 09 21:43:25.738694 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.738657    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpmsc\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-kube-api-access-qpmsc\") pod \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") "
Sep 09 21:43:25.738773 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.738738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-ca-trust-extracted\") pod \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") "
Sep 09 21:43:25.738936 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.738907    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "11278b72-85bd-4ca1-9ce0-904d0aa6ad00" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:25.739464 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.739416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "11278b72-85bd-4ca1-9ce0-904d0aa6ad00" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:25.739589 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.738769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-certificates\") pod \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") "
Sep 09 21:43:25.739589 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.739521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-bound-sa-token\") pod \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\" (UID: \"11278b72-85bd-4ca1-9ce0-904d0aa6ad00\") "
Sep 09 21:43:25.740048 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.739760    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-trusted-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:25.740048 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.739782    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-certificates\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:25.741128 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.741079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "11278b72-85bd-4ca1-9ce0-904d0aa6ad00" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:43:25.741235 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.741206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "11278b72-85bd-4ca1-9ce0-904d0aa6ad00" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:43:25.741294 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.741229    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "11278b72-85bd-4ca1-9ce0-904d0aa6ad00" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:43:25.741294 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.741242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-kube-api-access-qpmsc" (OuterVolumeSpecName: "kube-api-access-qpmsc") pod "11278b72-85bd-4ca1-9ce0-904d0aa6ad00" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00"). InnerVolumeSpecName "kube-api-access-qpmsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:43:25.741987 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.741964    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "11278b72-85bd-4ca1-9ce0-904d0aa6ad00" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:43:25.747444 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.747419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "11278b72-85bd-4ca1-9ce0-904d0aa6ad00" (UID: "11278b72-85bd-4ca1-9ce0-904d0aa6ad00"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:43:25.840558 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.840504    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-ca-trust-extracted\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:25.840558 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.840551    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-bound-sa-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:25.840558 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.840563    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-registry-tls\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:25.840558 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.840571    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-installation-pull-secrets\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:25.840858 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.840581    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-image-registry-private-configuration\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:25.840858 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.840591    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpmsc\" (UniqueName: \"kubernetes.io/projected/11278b72-85bd-4ca1-9ce0-904d0aa6ad00-kube-api-access-qpmsc\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:25.936845 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.936814    2579 generic.go:356] "Generic (PLEG): container finished" podID="11278b72-85bd-4ca1-9ce0-904d0aa6ad00" containerID="2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be" exitCode=0
Sep 09 21:43:25.936995 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.936876    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd"
Sep 09 21:43:25.936995 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.936903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd" event={"ID":"11278b72-85bd-4ca1-9ce0-904d0aa6ad00","Type":"ContainerDied","Data":"2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be"}
Sep 09 21:43:25.936995 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.936983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bc57fd5bc-rfskd" event={"ID":"11278b72-85bd-4ca1-9ce0-904d0aa6ad00","Type":"ContainerDied","Data":"67ee295a1146109a07a6609c89a547fb44f0efe7339cd441afd8c1c04dca004e"}
Sep 09 21:43:25.937142 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.937008    2579 scope.go:117] "RemoveContainer" containerID="2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be"
Sep 09 21:43:25.937142 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.937123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v99wb"
Sep 09 21:43:25.945723 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.945709    2579 scope.go:117] "RemoveContainer" containerID="2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be"
Sep 09 21:43:25.945951 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:25.945932    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be\": container with ID starting with 2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be not found: ID does not exist" containerID="2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be"
Sep 09 21:43:25.946012 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.945958    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be"} err="failed to get container status \"2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be\": rpc error: code = NotFound desc = could not find container \"2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be\": container with ID starting with 2133566408f6d8aa0bc7a83854ea85844912e0beae24e9d0cf011a94f5f5b7be not found: ID does not exist"
Sep 09 21:43:25.958300 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.958277    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bc57fd5bc-rfskd"]
Sep 09 21:43:25.963758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:25.963739    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bc57fd5bc-rfskd"]
Sep 09 21:43:26.092173 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:26.092088    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11278b72-85bd-4ca1-9ce0-904d0aa6ad00" path="/var/lib/kubelet/pods/11278b72-85bd-4ca1-9ce0-904d0aa6ad00/volumes"
Sep 09 21:43:28.353182 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.353148    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"]
Sep 09 21:43:28.353537 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.353373    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11278b72-85bd-4ca1-9ce0-904d0aa6ad00" containerName="registry"
Sep 09 21:43:28.353537 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.353382    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11278b72-85bd-4ca1-9ce0-904d0aa6ad00" containerName="registry"
Sep 09 21:43:28.353537 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.353432    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11278b72-85bd-4ca1-9ce0-904d0aa6ad00" containerName="registry"
Sep 09 21:43:28.357161 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.357144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.359728 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.359701    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 21:43:28.360565 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.360545    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 21:43:28.360675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.360561    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 21:43:28.360675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.360625    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 21:43:28.360675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.360649    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 21:43:28.360830 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.360677    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6xx78\""
Sep 09 21:43:28.367224 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.367199    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"]
Sep 09 21:43:28.371537 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.371515    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mnj2k"]
Sep 09 21:43:28.375333 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.375316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.378300 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.378282    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 21:43:28.378428 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.378301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 21:43:28.378428 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.378422    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-258df\""
Sep 09 21:43:28.378563 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.378546    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 21:43:28.385749 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.385730    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"]
Sep 09 21:43:28.392672 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.392653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.394531 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.394513    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 21:43:28.394915 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.394890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 21:43:28.395021 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.394931    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gvbtr\""
Sep 09 21:43:28.395021 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.394950    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 21:43:28.406329 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.406308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"]
Sep 09 21:43:28.460588 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-tls\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-sys\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4441a240-ee74-4b9f-906c-318581bb2864-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c136aec1-2cab-45a6-aacc-141254796986-metrics-client-ca\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbktl\" (UniqueName: \"kubernetes.io/projected/c136aec1-2cab-45a6-aacc-141254796986-kube-api-access-fbktl\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-textfile\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.460796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xc46\" (UniqueName: \"kubernetes.io/projected/4441a240-ee74-4b9f-906c-318581bb2864-kube-api-access-6xc46\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdsp\" (UniqueName: \"kubernetes.io/projected/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-api-access-xrdsp\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-wtmp\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-root\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.461138 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.460988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.561892 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.561855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.561892 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.561896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-textfile\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562123 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.561916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.562123 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:28.562002    2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 21:43:28.562123 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.562123 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:28.562064    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-tls podName:4441a240-ee74-4b9f-906c-318581bb2864 nodeName:}" failed. No retries permitted until 2026-09-09 21:43:29.062047556 +0000 UTC m=+283.484410235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-fp486" (UID: "4441a240-ee74-4b9f-906c-318581bb2864") : secret "openshift-state-metrics-tls" not found
Sep 09 21:43:28.562123 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.562123 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:28.562100    2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xc46\" (UniqueName: \"kubernetes.io/projected/4441a240-ee74-4b9f-906c-318581bb2864-kube-api-access-6xc46\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:28.562145    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-tls podName:31b6d756-4bd2-4f8b-8abc-8fbc000a5329 nodeName:}" failed. No retries permitted until 2026-09-09 21:43:29.06212834 +0000 UTC m=+283.484491039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-mh96n" (UID: "31b6d756-4bd2-4f8b-8abc-8fbc000a5329") : secret "kube-state-metrics-tls" not found
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdsp\" (UniqueName: \"kubernetes.io/projected/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-api-access-xrdsp\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-wtmp\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-textfile\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-root\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-tls\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-sys\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-wtmp\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.562439 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4441a240-ee74-4b9f-906c-318581bb2864-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c136aec1-2cab-45a6-aacc-141254796986-metrics-client-ca\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-root\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbktl\" (UniqueName: \"kubernetes.io/projected/c136aec1-2cab-45a6-aacc-141254796986-kube-api-access-fbktl\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:28.562523    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:28.562581    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-tls podName:c136aec1-2cab-45a6-aacc-141254796986 nodeName:}" failed. No retries permitted until 2026-09-09 21:43:29.062565462 +0000 UTC m=+283.484928147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-tls") pod "node-exporter-mnj2k" (UID: "c136aec1-2cab-45a6-aacc-141254796986") : secret "node-exporter-tls" not found
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.562951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c136aec1-2cab-45a6-aacc-141254796986-sys\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.563021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.563035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.563196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.563093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.563766 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.563485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4441a240-ee74-4b9f-906c-318581bb2864-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.563766 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.563587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c136aec1-2cab-45a6-aacc-141254796986-metrics-client-ca\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.564796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.564769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.564898 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.564772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.564898 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.564809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:28.573817 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.573779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbktl\" (UniqueName: \"kubernetes.io/projected/c136aec1-2cab-45a6-aacc-141254796986-kube-api-access-fbktl\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:28.574013 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.573994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xc46\" (UniqueName: \"kubernetes.io/projected/4441a240-ee74-4b9f-906c-318581bb2864-kube-api-access-6xc46\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:28.574154 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:28.574137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdsp\" (UniqueName: \"kubernetes.io/projected/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-api-access-xrdsp\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:29.067284 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.067248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-tls\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:29.067462 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.067307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:29.067462 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.067333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:29.069755 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.069720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c136aec1-2cab-45a6-aacc-141254796986-node-exporter-tls\") pod \"node-exporter-mnj2k\" (UID: \"c136aec1-2cab-45a6-aacc-141254796986\") " pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:29.069889 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.069765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31b6d756-4bd2-4f8b-8abc-8fbc000a5329-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mh96n\" (UID: \"31b6d756-4bd2-4f8b-8abc-8fbc000a5329\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:29.069889 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.069777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4441a240-ee74-4b9f-906c-318581bb2864-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fp486\" (UID: \"4441a240-ee74-4b9f-906c-318581bb2864\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:29.266120 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.266080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"
Sep 09 21:43:29.284202 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.284168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mnj2k"
Sep 09 21:43:29.294419 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:29.294383    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc136aec1_2cab_45a6_aacc_141254796986.slice/crio-9dc329ae792285de59f2730accb4f2269fd2a01001c49c73c0b5b8ef964e028a WatchSource:0}: Error finding container 9dc329ae792285de59f2730accb4f2269fd2a01001c49c73c0b5b8ef964e028a: Status 404 returned error can't find the container with id 9dc329ae792285de59f2730accb4f2269fd2a01001c49c73c0b5b8ef964e028a
Sep 09 21:43:29.300808 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.300782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"
Sep 09 21:43:29.435900 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.435865    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 21:43:29.440069 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.440051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.442312 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.442286    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ktmmt\""
Sep 09 21:43:29.442441 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.442399    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 21:43:29.442441 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.442423    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 21:43:29.442551 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.442526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 21:43:29.442657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.442641    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 21:43:29.442755 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.442724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 21:43:29.442866 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.442854    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 21:43:29.443335 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.443317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 21:43:29.443412 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.443318    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 21:43:29.447928 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.447815    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 21:43:29.455259 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.455236    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 21:43:29.470752 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.470726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.470870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.470771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:43:29.470870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.470808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.470870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.470836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.470973 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.470943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.470981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmk6\" (UniqueName: \"kubernetes.io/projected/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-kube-api-access-swmk6\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471041 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.471007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471085 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.471063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471135 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.471116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471171 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.471146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471250 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.471186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471250 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.471216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471250 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.471231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.471250 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.471245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.473659 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.473638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d834a708-778c-44a5-ad5a-ecdcb56aa53f-metrics-tls\") pod \"dns-default-v99wb\" (UID: \"d834a708-778c-44a5-ad5a-ecdcb56aa53f\") " pod="openshift-dns/dns-default-v99wb"
Sep 09 21:43:29.539574 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.539550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fp486"]
Sep 09 21:43:29.540010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.539993    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8h77q\""
Sep 09 21:43:29.542058 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:29.542034    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4441a240_ee74_4b9f_906c_318581bb2864.slice/crio-8de6748730a0730a3bd12e44b31e6b235fe5d49085e3d2782dd5a960d89e945d WatchSource:0}: Error finding container 8de6748730a0730a3bd12e44b31e6b235fe5d49085e3d2782dd5a960d89e945d: Status 404 returned error can't find the container with id 8de6748730a0730a3bd12e44b31e6b235fe5d49085e3d2782dd5a960d89e945d
Sep 09 21:43:29.549139 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.549119    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v99wb"
Sep 09 21:43:29.572680 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.572830 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.572830 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.572830 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.572830 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmk6\" (UniqueName: \"kubernetes.io/projected/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-kube-api-access-swmk6\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.572975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.573014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.573040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.573341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.573106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.574485 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.574461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.577580 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.576384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.577580 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.576966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-web-config\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.577580 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.577215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.577580 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.577344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.577947 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.577905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.579661 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.578987    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mh96n"]
Sep 09 21:43:29.579661 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.579264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.580106 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.580063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-config-out\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.580495 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.580473    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.580686 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.580669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.581006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.580987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.581433 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.581409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmk6\" (UniqueName: \"kubernetes.io/projected/9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c-kube-api-access-swmk6\") pod \"alertmanager-main-0\" (UID: \"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.582039 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:29.582008    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b6d756_4bd2_4f8b_8abc_8fbc000a5329.slice/crio-ac3696e1d2c1a4e6d387471980118ad89d338676ded8d0bd4ebd40d60e75cd52 WatchSource:0}: Error finding container ac3696e1d2c1a4e6d387471980118ad89d338676ded8d0bd4ebd40d60e75cd52: Status 404 returned error can't find the container with id ac3696e1d2c1a4e6d387471980118ad89d338676ded8d0bd4ebd40d60e75cd52
Sep 09 21:43:29.750497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.750461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 21:43:29.824568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.824540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v99wb"]
Sep 09 21:43:29.826997 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:29.826949    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd834a708_778c_44a5_ad5a_ecdcb56aa53f.slice/crio-c4d11d505473580f8ff4fa552260e4ff08bdc32f0c95ce4f3bbf74388bcbf6bb WatchSource:0}: Error finding container c4d11d505473580f8ff4fa552260e4ff08bdc32f0c95ce4f3bbf74388bcbf6bb: Status 404 returned error can't find the container with id c4d11d505473580f8ff4fa552260e4ff08bdc32f0c95ce4f3bbf74388bcbf6bb
Sep 09 21:43:29.953824 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.953782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486" event={"ID":"4441a240-ee74-4b9f-906c-318581bb2864","Type":"ContainerStarted","Data":"661f56ee1cb854a94a890d5576be181fe1ded8d6133742bc5132294fb607aa29"}
Sep 09 21:43:29.953982 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.953858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486" event={"ID":"4441a240-ee74-4b9f-906c-318581bb2864","Type":"ContainerStarted","Data":"3d4f629628b68b5e939cbe4f2d1b32d7f629a0f152a1145051f89a8497639eb6"}
Sep 09 21:43:29.953982 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.953887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486" event={"ID":"4441a240-ee74-4b9f-906c-318581bb2864","Type":"ContainerStarted","Data":"8de6748730a0730a3bd12e44b31e6b235fe5d49085e3d2782dd5a960d89e945d"}
Sep 09 21:43:29.954836 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.954814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v99wb" event={"ID":"d834a708-778c-44a5-ad5a-ecdcb56aa53f","Type":"ContainerStarted","Data":"c4d11d505473580f8ff4fa552260e4ff08bdc32f0c95ce4f3bbf74388bcbf6bb"}
Sep 09 21:43:29.955984 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.955953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n" event={"ID":"31b6d756-4bd2-4f8b-8abc-8fbc000a5329","Type":"ContainerStarted","Data":"ac3696e1d2c1a4e6d387471980118ad89d338676ded8d0bd4ebd40d60e75cd52"}
Sep 09 21:43:29.957065 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:29.957044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnj2k" event={"ID":"c136aec1-2cab-45a6-aacc-141254796986","Type":"ContainerStarted","Data":"9dc329ae792285de59f2730accb4f2269fd2a01001c49c73c0b5b8ef964e028a"}
Sep 09 21:43:30.032653 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:30.032497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 21:43:30.035885 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:30.035854    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa8dee8_61a9_4e2a_a7d4_8fc41fbfa01c.slice/crio-78c54124b4af6dba3d47f08511e64eb2415c0950bc0b2769ab0fdd251304c520 WatchSource:0}: Error finding container 78c54124b4af6dba3d47f08511e64eb2415c0950bc0b2769ab0fdd251304c520: Status 404 returned error can't find the container with id 78c54124b4af6dba3d47f08511e64eb2415c0950bc0b2769ab0fdd251304c520
Sep 09 21:43:30.964729 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:30.964585    2579 generic.go:356] "Generic (PLEG): container finished" podID="c136aec1-2cab-45a6-aacc-141254796986" containerID="d02520921a062b838f49809ae8a28a53bcf57d9c4c9ec0401b6bcf27d9bf9cb7" exitCode=0
Sep 09 21:43:30.964729 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:30.964671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnj2k" event={"ID":"c136aec1-2cab-45a6-aacc-141254796986","Type":"ContainerDied","Data":"d02520921a062b838f49809ae8a28a53bcf57d9c4c9ec0401b6bcf27d9bf9cb7"}
Sep 09 21:43:30.966755 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:30.966692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c","Type":"ContainerStarted","Data":"78c54124b4af6dba3d47f08511e64eb2415c0950bc0b2769ab0fdd251304c520"}
Sep 09 21:43:31.973304 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:31.973265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486" event={"ID":"4441a240-ee74-4b9f-906c-318581bb2864","Type":"ContainerStarted","Data":"bc14dda1d4f57c58befad64548d7d61f02ca5b317254dbdb858b4fc49390a72c"}
Sep 09 21:43:31.975686 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:31.975640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n" event={"ID":"31b6d756-4bd2-4f8b-8abc-8fbc000a5329","Type":"ContainerStarted","Data":"edb8f8da2d4151f6e912f85b08f2aa03eb11e68398b5c6422ce555e8968de917"}
Sep 09 21:43:31.975837 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:31.975713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n" event={"ID":"31b6d756-4bd2-4f8b-8abc-8fbc000a5329","Type":"ContainerStarted","Data":"6a37abed5c7c6ca3f8fc40c373723ab68c696f78dada87311e9de2d7da19cbb8"}
Sep 09 21:43:31.975837 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:31.975747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n" event={"ID":"31b6d756-4bd2-4f8b-8abc-8fbc000a5329","Type":"ContainerStarted","Data":"a03ba8e5ded412b8a32c8b0a6758bdf9cd65d6ae67b08605043a27d53e5cf689"}
Sep 09 21:43:31.978144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:31.978123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnj2k" event={"ID":"c136aec1-2cab-45a6-aacc-141254796986","Type":"ContainerStarted","Data":"a363e69f44e67f816bb0955c6cd004220a5e89f5174d2bfed6e5ca9e9a0c0567"}
Sep 09 21:43:31.978282 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:31.978160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnj2k" event={"ID":"c136aec1-2cab-45a6-aacc-141254796986","Type":"ContainerStarted","Data":"406113d0bed94be2a8d18e84b756728148d276ff89958674e32469b480beac70"}
Sep 09 21:43:31.993316 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:31.993246    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fp486" podStartSLOduration=2.549510324 podStartE2EDuration="3.993230307s" totalImagesPullingTime="1.443719983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:31.992991871 +0000 UTC m=+286.415354572" watchObservedRunningTime="2026-09-09 21:43:31.993230307 +0000 UTC m=+286.415593006"
Sep 09 21:43:32.013470 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.013409    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mnj2k" podStartSLOduration=2.887576444 podStartE2EDuration="4.013386806s" totalImagesPullingTime="1.057646578s" totalInitContainerRuntime="68.163784ms" isStatefulPod=true podCreationTimestamp="2026-09-09 21:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:32.012672044 +0000 UTC m=+286.435034745" watchObservedRunningTime="2026-09-09 21:43:32.013386806 +0000 UTC m=+286.435749507"
Sep 09 21:43:32.032483 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.031467    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mh96n" podStartSLOduration=2.478226884 podStartE2EDuration="4.031445901s" totalImagesPullingTime="1.553219017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:32.030155375 +0000 UTC m=+286.452518089" watchObservedRunningTime="2026-09-09 21:43:32.031445901 +0000 UTC m=+286.453808601"
Sep 09 21:43:32.687176 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.687141    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"]
Sep 09 21:43:32.690211 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.690196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.695319 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.695297    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 21:43:32.696145 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.696111    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-430r559p42nhl\""
Sep 09 21:43:32.696145 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.696129    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kml6j\""
Sep 09 21:43:32.696305 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.696149    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 21:43:32.696305 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.696178    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 21:43:32.696305 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.696120    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 21:43:32.703618 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.701018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"]
Sep 09 21:43:32.800315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.800260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-audit-log\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.800497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.800333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.800497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.800395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-secret-metrics-server-client-certs\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.800497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.800430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-metrics-server-audit-profiles\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.800497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.800446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-client-ca-bundle\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.800497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.800465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rwr\" (UniqueName: \"kubernetes.io/projected/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-kube-api-access-f9rwr\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.800687 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.800531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-secret-metrics-server-tls\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.901237 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.901192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rwr\" (UniqueName: \"kubernetes.io/projected/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-kube-api-access-f9rwr\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.901452 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.901250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-secret-metrics-server-tls\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.901452 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.901324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-audit-log\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.901452 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.901386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.901452 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.901439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-secret-metrics-server-client-certs\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.901685 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.901472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-metrics-server-audit-profiles\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.901685 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.901498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-client-ca-bundle\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.901873 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.901845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-audit-log\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.902308 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.902280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.902482 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.902460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-metrics-server-audit-profiles\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.903938 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.903909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-secret-metrics-server-tls\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.904032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.903966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-client-ca-bundle\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.904077 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.904031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-secret-metrics-server-client-certs\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.909459 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.909435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rwr\" (UniqueName: \"kubernetes.io/projected/d2b24c70-d39c-40c7-ad04-8e3d4de8d35b-kube-api-access-f9rwr\") pod \"metrics-server-7bb65f78d5-2g9dl\" (UID: \"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b\") " pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:32.984112 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.984073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v99wb" event={"ID":"d834a708-778c-44a5-ad5a-ecdcb56aa53f","Type":"ContainerStarted","Data":"a0623a877052aa117c2e531271b13f90d9f0ae52101ba93d9e0c55d0fbb6db60"}
Sep 09 21:43:32.984556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.984133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v99wb" event={"ID":"d834a708-778c-44a5-ad5a-ecdcb56aa53f","Type":"ContainerStarted","Data":"d036167284b9a92c02052176e2c86b2702764b67cdb3f0e9cabbc69eec5e208d"}
Sep 09 21:43:32.984556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.984198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v99wb"
Sep 09 21:43:32.985858 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.985824    2579 generic.go:356] "Generic (PLEG): container finished" podID="9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c" containerID="f33991f76295db71ec69c891623c885e7dd6faa7f1da5f0c25281dd039476890" exitCode=0
Sep 09 21:43:32.985984 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:32.985898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c","Type":"ContainerDied","Data":"f33991f76295db71ec69c891623c885e7dd6faa7f1da5f0c25281dd039476890"}
Sep 09 21:43:33.000269 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.000245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:33.001959 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.001910    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v99wb" podStartSLOduration=251.636866745 podStartE2EDuration="4m14.001892279s" totalImagesPullingTime="2.365025534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:33.000813583 +0000 UTC m=+287.423176286" watchObservedRunningTime="2026-09-09 21:43:33.001892279 +0000 UTC m=+287.424254981"
Sep 09 21:43:33.170525 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.170447    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d69696db7-slb5n"]
Sep 09 21:43:33.201446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.201413    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf498676b-drwsv"]
Sep 09 21:43:33.205979 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.205959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.212280 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.212258    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 21:43:33.215863 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.215829    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf498676b-drwsv"]
Sep 09 21:43:33.271467 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.271309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"]
Sep 09 21:43:33.273462 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:33.273435    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b24c70_d39c_40c7_ad04_8e3d4de8d35b.slice/crio-dc0897080db57a9fc1f2adec81b54920dd114deb80640d92bb21a35aec91fe77 WatchSource:0}: Error finding container dc0897080db57a9fc1f2adec81b54920dd114deb80640d92bb21a35aec91fe77: Status 404 returned error can't find the container with id dc0897080db57a9fc1f2adec81b54920dd114deb80640d92bb21a35aec91fe77
Sep 09 21:43:33.304640 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.304591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-service-ca\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.304735 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.304655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-oauth-serving-cert\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.304735 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.304679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzh5\" (UniqueName: \"kubernetes.io/projected/001d00d5-10a9-42e7-bbe6-b48a7aa89994-kube-api-access-mhzh5\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.304735 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.304701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-oauth-config\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.304735 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.304720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-trusted-ca-bundle\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.304945 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.304797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-config\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.304945 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.304832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-serving-cert\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.405706 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.405674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-oauth-serving-cert\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.405706 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.405711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzh5\" (UniqueName: \"kubernetes.io/projected/001d00d5-10a9-42e7-bbe6-b48a7aa89994-kube-api-access-mhzh5\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.405953 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.405841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-oauth-config\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.405953 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.405882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-trusted-ca-bundle\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.405953 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.405937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-config\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.406103 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.405973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-serving-cert\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.406103 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.406057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-service-ca\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.406493 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.406458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-oauth-serving-cert\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.406649 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.406598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-config\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.406725 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.406679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-service-ca\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.406986 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.406954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-trusted-ca-bundle\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.408401 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.408369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-oauth-config\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.408520 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.408492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-serving-cert\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.412549 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.412529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzh5\" (UniqueName: \"kubernetes.io/projected/001d00d5-10a9-42e7-bbe6-b48a7aa89994-kube-api-access-mhzh5\") pod \"console-7bf498676b-drwsv\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") " pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.516895 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.516814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:33.787417 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.787326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf498676b-drwsv"]
Sep 09 21:43:33.791978 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:33.791931    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001d00d5_10a9_42e7_bbe6_b48a7aa89994.slice/crio-d0c9e14a7955edfdc6f8a1d6dd5c3eebfb0dfdc26406cbacbbb660aa4a889970 WatchSource:0}: Error finding container d0c9e14a7955edfdc6f8a1d6dd5c3eebfb0dfdc26406cbacbbb660aa4a889970: Status 404 returned error can't find the container with id d0c9e14a7955edfdc6f8a1d6dd5c3eebfb0dfdc26406cbacbbb660aa4a889970
Sep 09 21:43:33.993010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.992978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf498676b-drwsv" event={"ID":"001d00d5-10a9-42e7-bbe6-b48a7aa89994","Type":"ContainerStarted","Data":"6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582"}
Sep 09 21:43:33.993452 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.993031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf498676b-drwsv" event={"ID":"001d00d5-10a9-42e7-bbe6-b48a7aa89994","Type":"ContainerStarted","Data":"d0c9e14a7955edfdc6f8a1d6dd5c3eebfb0dfdc26406cbacbbb660aa4a889970"}
Sep 09 21:43:33.994456 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:33.994428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl" event={"ID":"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b","Type":"ContainerStarted","Data":"dc0897080db57a9fc1f2adec81b54920dd114deb80640d92bb21a35aec91fe77"}
Sep 09 21:43:34.008838 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.008790    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf498676b-drwsv" podStartSLOduration=1.008768461 podStartE2EDuration="1.008768461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:34.008755702 +0000 UTC m=+288.431118405" watchObservedRunningTime="2026-09-09 21:43:34.008768461 +0000 UTC m=+288.431131162"
Sep 09 21:43:34.669745 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.669708    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 21:43:34.674227 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.674203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.676687 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.676654    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 21:43:34.676797 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.676654    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 21:43:34.676797 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.676656    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 21:43:34.676904 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.676667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 21:43:34.677690 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.677451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 21:43:34.677690 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.677477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-evvmpr8t327s8\""
Sep 09 21:43:34.677690 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.677530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 21:43:34.677690 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.677555    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 21:43:34.677690 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.677591    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 21:43:34.678032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.677809    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 21:43:34.678032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.678005    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c6nzx\""
Sep 09 21:43:34.678032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.678017    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 21:43:34.678419 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.678388    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 21:43:34.678537 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.678460    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 21:43:34.680717 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.680697    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 21:43:34.683479 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.683239    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 21:43:34.691791 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.691767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 21:43:34.718823 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.718796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3969580-afdb-4791-9790-5cddd3e6d160-config-out\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.718920 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.718875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-config\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.718920 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.718904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719035 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.718953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3969580-afdb-4791-9790-5cddd3e6d160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719035 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.718970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719035 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719035 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719227 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719227 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719227 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719227 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719227 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-web-config\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719437 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719437 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719437 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719437 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719437 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719685 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719685 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csd5d\" (UniqueName: \"kubernetes.io/projected/d3969580-afdb-4791-9790-5cddd3e6d160-kube-api-access-csd5d\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.719685 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.719509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821056 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821194 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821194 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821194 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821194 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csd5d\" (UniqueName: \"kubernetes.io/projected/d3969580-afdb-4791-9790-5cddd3e6d160-kube-api-access-csd5d\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821194 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821359 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3969580-afdb-4791-9790-5cddd3e6d160-config-out\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821359 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-config\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821359 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821359 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3969580-afdb-4791-9790-5cddd3e6d160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821359 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821359 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821359 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-web-config\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821771 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.821771 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.821579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.822507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.824040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.824288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.825134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.825254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.825509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.825542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.825511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826783 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.826336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3969580-afdb-4791-9790-5cddd3e6d160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826783 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.826716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.826783 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.826768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3969580-afdb-4791-9790-5cddd3e6d160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.827926 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.827899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.828058 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.827998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3969580-afdb-4791-9790-5cddd3e6d160-config-out\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.828330 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.828305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-web-config\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.828988 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.828967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.829250 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.829231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.830270 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.830248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-config\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.831290 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.831265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.832383 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.832334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3969580-afdb-4791-9790-5cddd3e6d160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.832777 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.832756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csd5d\" (UniqueName: \"kubernetes.io/projected/d3969580-afdb-4791-9790-5cddd3e6d160-kube-api-access-csd5d\") pod \"prometheus-k8s-0\" (UID: \"d3969580-afdb-4791-9790-5cddd3e6d160\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:34.987548 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:34.987455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:35.000948 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:35.000914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl" event={"ID":"d2b24c70-d39c-40c7-ad04-8e3d4de8d35b","Type":"ContainerStarted","Data":"c42fb7806241934681583e0b4b4cc85b24928ca5b0fb99a4aba37275a3052fae"}
Sep 09 21:43:35.016702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:35.016638    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl" podStartSLOduration=1.585461518 podStartE2EDuration="3.016615249s" totalImagesPullingTime="1.431153731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:35.016310007 +0000 UTC m=+289.438672708" watchObservedRunningTime="2026-09-09 21:43:35.016615249 +0000 UTC m=+289.438977944"
Sep 09 21:43:35.409881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:35.409847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 21:43:35.414779 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:43:35.414745    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3969580_afdb_4791_9790_5cddd3e6d160.slice/crio-3a84378f3cd742c4f745282209c9b91d4995db2acfe85569a2def9f2efd677d9 WatchSource:0}: Error finding container 3a84378f3cd742c4f745282209c9b91d4995db2acfe85569a2def9f2efd677d9: Status 404 returned error can't find the container with id 3a84378f3cd742c4f745282209c9b91d4995db2acfe85569a2def9f2efd677d9
Sep 09 21:43:36.007737 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:36.007702    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3969580-afdb-4791-9790-5cddd3e6d160" containerID="889ec510c62d1da20c527829888a817732415912f022ae7e3810a15ec331175b" exitCode=0
Sep 09 21:43:36.008181 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:36.007820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3969580-afdb-4791-9790-5cddd3e6d160","Type":"ContainerDied","Data":"889ec510c62d1da20c527829888a817732415912f022ae7e3810a15ec331175b"}
Sep 09 21:43:36.008181 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:36.007925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3969580-afdb-4791-9790-5cddd3e6d160","Type":"ContainerStarted","Data":"3a84378f3cd742c4f745282209c9b91d4995db2acfe85569a2def9f2efd677d9"}
Sep 09 21:43:36.011692 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:36.011657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c","Type":"ContainerStarted","Data":"37aa9c6564074ec13155934fa061145fa7e624075b66462a1c512aa330598eab"}
Sep 09 21:43:36.011802 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:36.011729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c","Type":"ContainerStarted","Data":"d795f1a892b487f2013299bf1dccd875e56ec2ea4ddf8141c6e6f2a8913fda96"}
Sep 09 21:43:36.011802 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:36.011756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c","Type":"ContainerStarted","Data":"ba1a9d979d2d690a216eb21142979477f81cae80607e9f6a5acc308fe2c94a07"}
Sep 09 21:43:36.011802 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:36.011782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c","Type":"ContainerStarted","Data":"4e2d1aaea1da3dc60be63e7722304bcc7504a983dfc461b06a0d62971514f040"}
Sep 09 21:43:36.011979 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:36.011816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c","Type":"ContainerStarted","Data":"8d44de659250b09b058627f32e87de2bc6ad3629e6500ab163563e7e3c2cd058"}
Sep 09 21:43:37.018734 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:37.018700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c","Type":"ContainerStarted","Data":"dc572a89e9b85bef3b0cc0f327293129505dfb17e408bee928c906d8b2850576"}
Sep 09 21:43:37.045538 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:37.045470    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.019091655 podStartE2EDuration="8.045445367s" totalImagesPullingTime="5.016343672s" totalInitContainerRuntime="10.01004ms" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:37.043250088 +0000 UTC m=+291.465612790" watchObservedRunningTime="2026-09-09 21:43:37.045445367 +0000 UTC m=+291.467808069"
Sep 09 21:43:40.037160 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:40.037120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3969580-afdb-4791-9790-5cddd3e6d160","Type":"ContainerStarted","Data":"715c9171381e13fcb99a74686f2605fdff006675286a40ad44a0751de2ac3828"}
Sep 09 21:43:40.037618 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:40.037197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3969580-afdb-4791-9790-5cddd3e6d160","Type":"ContainerStarted","Data":"6603580fc886572e31a26bbc70c4528c42c16753ec53d70982e6a5c8a4aafea8"}
Sep 09 21:43:42.049551 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:42.049511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3969580-afdb-4791-9790-5cddd3e6d160","Type":"ContainerStarted","Data":"42f0613255171418df2fc9d19736b5b2af06e6962c07955760ec78aa866e5da0"}
Sep 09 21:43:42.049969 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:42.049594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3969580-afdb-4791-9790-5cddd3e6d160","Type":"ContainerStarted","Data":"5c412c818a04f5f11664c94b22b439ed02b330fce1ce233f7d0e4a74e6ff7760"}
Sep 09 21:43:42.049969 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:42.049660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3969580-afdb-4791-9790-5cddd3e6d160","Type":"ContainerStarted","Data":"cb5d492dab7bad1dafb278ac36209c9bdbcc5deb684e13c38fe5f2a4d6870089"}
Sep 09 21:43:42.049969 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:42.049698    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3969580-afdb-4791-9790-5cddd3e6d160","Type":"ContainerStarted","Data":"f31cdf12fe0b69519c8571e495b4db1bc2dfd37e31fe7e59b36bf73bdd199815"}
Sep 09 21:43:42.078273 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:42.078217    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.059781919 podStartE2EDuration="8.078197982s" totalImagesPullingTime="4.997224049s" totalInitContainerRuntime="21.192014ms" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:42.074905517 +0000 UTC m=+296.497268217" watchObservedRunningTime="2026-09-09 21:43:42.078197982 +0000 UTC m=+296.500560681"
Sep 09 21:43:42.997215 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:42.997182    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v99wb"
Sep 09 21:43:43.517553 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:43.517510    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:43.518007 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:43.517861    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:43.522568 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:43.522546    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:44.061462 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:44.061429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:43:44.987724 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:44.987686    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:43:45.998461 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:45.998436    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 21:43:53.001407 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:53.001373    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:53.004259 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:53.001438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:43:58.190221 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.190157    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d69696db7-slb5n" podUID="100f3947-ee9c-42c4-98a8-a6d578af512d" containerName="console" containerID="cri-o://9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4" gracePeriod=15
Sep 09 21:43:58.439654 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.439629    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:58.538947 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.538867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-oauth-serving-cert\") pod \"100f3947-ee9c-42c4-98a8-a6d578af512d\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") "
Sep 09 21:43:58.538947 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.538912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-oauth-config\") pod \"100f3947-ee9c-42c4-98a8-a6d578af512d\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") "
Sep 09 21:43:58.538947 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.538933    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-service-ca\") pod \"100f3947-ee9c-42c4-98a8-a6d578af512d\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") "
Sep 09 21:43:58.539208 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.539000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-serving-cert\") pod \"100f3947-ee9c-42c4-98a8-a6d578af512d\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") "
Sep 09 21:43:58.539208 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.539022    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2p65\" (UniqueName: \"kubernetes.io/projected/100f3947-ee9c-42c4-98a8-a6d578af512d-kube-api-access-b2p65\") pod \"100f3947-ee9c-42c4-98a8-a6d578af512d\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") "
Sep 09 21:43:58.539208 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.539041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-console-config\") pod \"100f3947-ee9c-42c4-98a8-a6d578af512d\" (UID: \"100f3947-ee9c-42c4-98a8-a6d578af512d\") "
Sep 09 21:43:58.539397 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.539361    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-service-ca" (OuterVolumeSpecName: "service-ca") pod "100f3947-ee9c-42c4-98a8-a6d578af512d" (UID: "100f3947-ee9c-42c4-98a8-a6d578af512d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:58.539397 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.539339    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "100f3947-ee9c-42c4-98a8-a6d578af512d" (UID: "100f3947-ee9c-42c4-98a8-a6d578af512d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:58.539579 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.539519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-console-config" (OuterVolumeSpecName: "console-config") pod "100f3947-ee9c-42c4-98a8-a6d578af512d" (UID: "100f3947-ee9c-42c4-98a8-a6d578af512d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:43:58.541331 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.541301    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "100f3947-ee9c-42c4-98a8-a6d578af512d" (UID: "100f3947-ee9c-42c4-98a8-a6d578af512d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:43:58.541548 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.541526    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100f3947-ee9c-42c4-98a8-a6d578af512d-kube-api-access-b2p65" (OuterVolumeSpecName: "kube-api-access-b2p65") pod "100f3947-ee9c-42c4-98a8-a6d578af512d" (UID: "100f3947-ee9c-42c4-98a8-a6d578af512d"). InnerVolumeSpecName "kube-api-access-b2p65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:43:58.541631 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.541560    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "100f3947-ee9c-42c4-98a8-a6d578af512d" (UID: "100f3947-ee9c-42c4-98a8-a6d578af512d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:43:58.640013 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.639964    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-serving-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:58.640013 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.640007    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2p65\" (UniqueName: \"kubernetes.io/projected/100f3947-ee9c-42c4-98a8-a6d578af512d-kube-api-access-b2p65\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:58.640013 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.640029    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-console-config\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:58.640256 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.640038    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-oauth-serving-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:58.640256 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.640047    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/100f3947-ee9c-42c4-98a8-a6d578af512d-console-oauth-config\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:58.640256 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:58.640055    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/100f3947-ee9c-42c4-98a8-a6d578af512d-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:43:59.125254 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.125221    2579 generic.go:356] "Generic (PLEG): container finished" podID="100f3947-ee9c-42c4-98a8-a6d578af512d" containerID="9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4" exitCode=0
Sep 09 21:43:59.125467 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.125279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d69696db7-slb5n" event={"ID":"100f3947-ee9c-42c4-98a8-a6d578af512d","Type":"ContainerDied","Data":"9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4"}
Sep 09 21:43:59.125467 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.125290    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d69696db7-slb5n"
Sep 09 21:43:59.125467 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.125336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d69696db7-slb5n" event={"ID":"100f3947-ee9c-42c4-98a8-a6d578af512d","Type":"ContainerDied","Data":"0a6d1587e3707273d07509fde058b536a3936e4f072c7786f9bbcd60cf948ff1"}
Sep 09 21:43:59.125467 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.125355    2579 scope.go:117] "RemoveContainer" containerID="9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4"
Sep 09 21:43:59.134301 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.134282    2579 scope.go:117] "RemoveContainer" containerID="9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4"
Sep 09 21:43:59.134562 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:43:59.134543    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4\": container with ID starting with 9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4 not found: ID does not exist" containerID="9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4"
Sep 09 21:43:59.134679 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.134571    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4"} err="failed to get container status \"9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4\": rpc error: code = NotFound desc = could not find container \"9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4\": container with ID starting with 9c40113198b96903b81764dea342919bf17f12d047eba613fc6c6400d382cdf4 not found: ID does not exist"
Sep 09 21:43:59.145796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.145772    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d69696db7-slb5n"]
Sep 09 21:43:59.149517 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:43:59.149493    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d69696db7-slb5n"]
Sep 09 21:44:00.091232 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:00.091192    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="100f3947-ee9c-42c4-98a8-a6d578af512d" path="/var/lib/kubelet/pods/100f3947-ee9c-42c4-98a8-a6d578af512d/volumes"
Sep 09 21:44:04.148000 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:04.147969    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0aa425c-e7ed-4feb-9610-55b9cf1ee45e" containerID="88008589dee5f3686702b763f15b2f80bafbb732bc7a09f695158a1f62a8c788" exitCode=0
Sep 09 21:44:04.148349 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:04.148047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6z657" event={"ID":"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e","Type":"ContainerDied","Data":"88008589dee5f3686702b763f15b2f80bafbb732bc7a09f695158a1f62a8c788"}
Sep 09 21:44:04.148388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:04.148364    2579 scope.go:117] "RemoveContainer" containerID="88008589dee5f3686702b763f15b2f80bafbb732bc7a09f695158a1f62a8c788"
Sep 09 21:44:04.831066 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:04.830964    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c55bbf95b-wdt7z_5613aec3-899d-4cbf-a406-7e1d4b0f5bd7/router/0.log"
Sep 09 21:44:05.008179 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:05.008151    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mfrlp_667ff566-22cb-4cdc-beb3-6bab86102ec5/serve-healthcheck-canary/0.log"
Sep 09 21:44:05.154051 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:05.154012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6z657" event={"ID":"c0aa425c-e7ed-4feb-9610-55b9cf1ee45e","Type":"ContainerStarted","Data":"4295d66a65c442f854106987f939f2ab96999561ee48bf48ff7272bd5d3a7538"}
Sep 09 21:44:05.155579 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:05.155555    2579 generic.go:356] "Generic (PLEG): container finished" podID="598d76a6-65f4-48d0-8deb-96c1db8c084a" containerID="120fcc22fefdd0b2d14e574a224aded881a9ad59a36ff675942efff760f3b43d" exitCode=0
Sep 09 21:44:05.155702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:05.155613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s" event={"ID":"598d76a6-65f4-48d0-8deb-96c1db8c084a","Type":"ContainerDied","Data":"120fcc22fefdd0b2d14e574a224aded881a9ad59a36ff675942efff760f3b43d"}
Sep 09 21:44:05.155894 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:05.155881    2579 scope.go:117] "RemoveContainer" containerID="120fcc22fefdd0b2d14e574a224aded881a9ad59a36ff675942efff760f3b43d"
Sep 09 21:44:06.161675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:06.161636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-25d5s" event={"ID":"598d76a6-65f4-48d0-8deb-96c1db8c084a","Type":"ContainerStarted","Data":"f3010d6c074ca1ed6cd2c9c4cf2e323b5858d43e38a687f946c2cd59f78cb05c"}
Sep 09 21:44:13.007294 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:13.007258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:44:13.011226 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:13.011203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bb65f78d5-2g9dl"
Sep 09 21:44:34.988449 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:34.988406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:44:35.004928 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:35.004882    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:44:35.302836 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:44:35.302761    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 21:45:17.095181 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:17.095142    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf498676b-drwsv"]
Sep 09 21:45:42.116977 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.116939    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf498676b-drwsv" podUID="001d00d5-10a9-42e7-bbe6-b48a7aa89994" containerName="console" containerID="cri-o://6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582" gracePeriod=15
Sep 09 21:45:42.362176 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.362152    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:45:42.458491 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.458392    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhzh5\" (UniqueName: \"kubernetes.io/projected/001d00d5-10a9-42e7-bbe6-b48a7aa89994-kube-api-access-mhzh5\") pod \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") "
Sep 09 21:45:42.458491 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.458430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-trusted-ca-bundle\") pod \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") "
Sep 09 21:45:42.458491 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.458479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-serving-cert\") pod \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") "
Sep 09 21:45:42.458811 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.458581    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-config\") pod \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") "
Sep 09 21:45:42.458811 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.458661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-service-ca\") pod \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") "
Sep 09 21:45:42.458811 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.458690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-oauth-serving-cert\") pod \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") "
Sep 09 21:45:42.458811 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.458758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-oauth-config\") pod \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\" (UID: \"001d00d5-10a9-42e7-bbe6-b48a7aa89994\") "
Sep 09 21:45:42.459016 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.458987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-config" (OuterVolumeSpecName: "console-config") pod "001d00d5-10a9-42e7-bbe6-b48a7aa89994" (UID: "001d00d5-10a9-42e7-bbe6-b48a7aa89994"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:45:42.459075 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.459017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-service-ca" (OuterVolumeSpecName: "service-ca") pod "001d00d5-10a9-42e7-bbe6-b48a7aa89994" (UID: "001d00d5-10a9-42e7-bbe6-b48a7aa89994"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:45:42.459075 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.459043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "001d00d5-10a9-42e7-bbe6-b48a7aa89994" (UID: "001d00d5-10a9-42e7-bbe6-b48a7aa89994"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:45:42.459175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.459130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "001d00d5-10a9-42e7-bbe6-b48a7aa89994" (UID: "001d00d5-10a9-42e7-bbe6-b48a7aa89994"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:45:42.460749 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.460725    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "001d00d5-10a9-42e7-bbe6-b48a7aa89994" (UID: "001d00d5-10a9-42e7-bbe6-b48a7aa89994"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:45:42.460749 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.460741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "001d00d5-10a9-42e7-bbe6-b48a7aa89994" (UID: "001d00d5-10a9-42e7-bbe6-b48a7aa89994"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:45:42.460890 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.460807    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001d00d5-10a9-42e7-bbe6-b48a7aa89994-kube-api-access-mhzh5" (OuterVolumeSpecName: "kube-api-access-mhzh5") pod "001d00d5-10a9-42e7-bbe6-b48a7aa89994" (UID: "001d00d5-10a9-42e7-bbe6-b48a7aa89994"). InnerVolumeSpecName "kube-api-access-mhzh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:45:42.551950 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.551916    2579 generic.go:356] "Generic (PLEG): container finished" podID="001d00d5-10a9-42e7-bbe6-b48a7aa89994" containerID="6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582" exitCode=0
Sep 09 21:45:42.552096 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.551968    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf498676b-drwsv"
Sep 09 21:45:42.552096 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.552003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf498676b-drwsv" event={"ID":"001d00d5-10a9-42e7-bbe6-b48a7aa89994","Type":"ContainerDied","Data":"6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582"}
Sep 09 21:45:42.552096 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.552075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf498676b-drwsv" event={"ID":"001d00d5-10a9-42e7-bbe6-b48a7aa89994","Type":"ContainerDied","Data":"d0c9e14a7955edfdc6f8a1d6dd5c3eebfb0dfdc26406cbacbbb660aa4a889970"}
Sep 09 21:45:42.552096 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.552096    2579 scope.go:117] "RemoveContainer" containerID="6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582"
Sep 09 21:45:42.559809 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.559787    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-config\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:45:42.559912 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.559813    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:45:42.559912 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.559825    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-oauth-serving-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:45:42.559912 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.559839    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-oauth-config\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:45:42.559912 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.559852    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzh5\" (UniqueName: \"kubernetes.io/projected/001d00d5-10a9-42e7-bbe6-b48a7aa89994-kube-api-access-mhzh5\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:45:42.559912 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.559864    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001d00d5-10a9-42e7-bbe6-b48a7aa89994-trusted-ca-bundle\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:45:42.559912 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.559876    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/001d00d5-10a9-42e7-bbe6-b48a7aa89994-console-serving-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:45:42.561720 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.561698    2579 scope.go:117] "RemoveContainer" containerID="6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582"
Sep 09 21:45:42.561986 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:45:42.561968    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582\": container with ID starting with 6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582 not found: ID does not exist" containerID="6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582"
Sep 09 21:45:42.562040 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.561993    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582"} err="failed to get container status \"6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582\": rpc error: code = NotFound desc = could not find container \"6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582\": container with ID starting with 6a5250c20c07b3375c36965139334dd492f3765c463da620f372a88ae2be2582 not found: ID does not exist"
Sep 09 21:45:42.572859 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.572835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf498676b-drwsv"]
Sep 09 21:45:42.576387 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:42.576368    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf498676b-drwsv"]
Sep 09 21:45:44.091921 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:45:44.091888    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001d00d5-10a9-42e7-bbe6-b48a7aa89994" path="/var/lib/kubelet/pods/001d00d5-10a9-42e7-bbe6-b48a7aa89994/volumes"
Sep 09 21:47:42.854305 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.854220    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"]
Sep 09 21:47:42.854870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.854685    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100f3947-ee9c-42c4-98a8-a6d578af512d" containerName="console"
Sep 09 21:47:42.854870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.854705    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="100f3947-ee9c-42c4-98a8-a6d578af512d" containerName="console"
Sep 09 21:47:42.854870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.854725    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001d00d5-10a9-42e7-bbe6-b48a7aa89994" containerName="console"
Sep 09 21:47:42.854870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.854733    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="001d00d5-10a9-42e7-bbe6-b48a7aa89994" containerName="console"
Sep 09 21:47:42.854870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.854825    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="100f3947-ee9c-42c4-98a8-a6d578af512d" containerName="console"
Sep 09 21:47:42.854870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.854840    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="001d00d5-10a9-42e7-bbe6-b48a7aa89994" containerName="console"
Sep 09 21:47:42.857859 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.857838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:42.860112 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.860066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 21:47:42.860225 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.860196    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 21:47:42.860225 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.860209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 21:47:42.860337 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.860250    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7txkq\""
Sep 09 21:47:42.861337 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.861321    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 21:47:42.876586 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.876561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"]
Sep 09 21:47:42.981824 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.981788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81f4008d-743b-4c71-a8b6-3c24545ab2a8-tmp\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:42.982015 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.981842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2p5\" (UniqueName: \"kubernetes.io/projected/81f4008d-743b-4c71-a8b6-3c24545ab2a8-kube-api-access-cb2p5\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:42.982015 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.981863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81f4008d-743b-4c71-a8b6-3c24545ab2a8-manifests\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:42.982015 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.981955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81f4008d-743b-4c71-a8b6-3c24545ab2a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:42.982126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:42.982015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81f4008d-743b-4c71-a8b6-3c24545ab2a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.083319 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.083279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81f4008d-743b-4c71-a8b6-3c24545ab2a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.083319 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.083327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81f4008d-743b-4c71-a8b6-3c24545ab2a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.083566 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.083406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81f4008d-743b-4c71-a8b6-3c24545ab2a8-tmp\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.083566 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.083452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2p5\" (UniqueName: \"kubernetes.io/projected/81f4008d-743b-4c71-a8b6-3c24545ab2a8-kube-api-access-cb2p5\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.083566 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.083479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81f4008d-743b-4c71-a8b6-3c24545ab2a8-manifests\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.083938 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.083907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81f4008d-743b-4c71-a8b6-3c24545ab2a8-manifests\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.085898 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.085867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81f4008d-743b-4c71-a8b6-3c24545ab2a8-tmp\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.086014 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.085939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81f4008d-743b-4c71-a8b6-3c24545ab2a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.086058 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.086046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81f4008d-743b-4c71-a8b6-3c24545ab2a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.092283 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.092258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2p5\" (UniqueName: \"kubernetes.io/projected/81f4008d-743b-4c71-a8b6-3c24545ab2a8-kube-api-access-cb2p5\") pod \"opendatahub-operator-controller-manager-56bc947947-gwfqd\" (UID: \"81f4008d-743b-4c71-a8b6-3c24545ab2a8\") " pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.169630 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.169516    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:43.463881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.463788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"]
Sep 09 21:47:43.467852 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:47:43.467820    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f4008d_743b_4c71_a8b6_3c24545ab2a8.slice/crio-d4a943a4309fcbc11d1ccab0a760baff7c25ba3b42beb43476ceb6346086adc3 WatchSource:0}: Error finding container d4a943a4309fcbc11d1ccab0a760baff7c25ba3b42beb43476ceb6346086adc3: Status 404 returned error can't find the container with id d4a943a4309fcbc11d1ccab0a760baff7c25ba3b42beb43476ceb6346086adc3
Sep 09 21:47:43.469842 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:43.469817    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:47:44.039813 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:44.039776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd" event={"ID":"81f4008d-743b-4c71-a8b6-3c24545ab2a8","Type":"ContainerStarted","Data":"d4a943a4309fcbc11d1ccab0a760baff7c25ba3b42beb43476ceb6346086adc3"}
Sep 09 21:47:47.062500 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:47.062459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd" event={"ID":"81f4008d-743b-4c71-a8b6-3c24545ab2a8","Type":"ContainerStarted","Data":"3e30c01184ce3cf4aaf70bb091f05260e49341083c2f448f99ebbe079ab0338c"}
Sep 09 21:47:48.068413 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:48.068372    2579 generic.go:356] "Generic (PLEG): container finished" podID="81f4008d-743b-4c71-a8b6-3c24545ab2a8" containerID="3e30c01184ce3cf4aaf70bb091f05260e49341083c2f448f99ebbe079ab0338c" exitCode=0
Sep 09 21:47:48.068871 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:48.068455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd" event={"ID":"81f4008d-743b-4c71-a8b6-3c24545ab2a8","Type":"ContainerDied","Data":"3e30c01184ce3cf4aaf70bb091f05260e49341083c2f448f99ebbe079ab0338c"}
Sep 09 21:47:49.074477 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:49.074439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd" event={"ID":"81f4008d-743b-4c71-a8b6-3c24545ab2a8","Type":"ContainerStarted","Data":"6a5dc90134a0bc501ac7299ad6c4b165c7e97312de3da3fdd31dc701a0bad07e"}
Sep 09 21:47:49.074992 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:49.074547    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:47:49.098056 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:47:49.097999    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd" podStartSLOduration=2.819471238 podStartE2EDuration="7.097979525s" totalImagesPullingTime="3.274420449s" totalInitContainerRuntime="1.004087838s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:49.097833999 +0000 UTC m=+543.520196700" watchObservedRunningTime="2026-09-09 21:47:49.097979525 +0000 UTC m=+543.520342225"
Sep 09 21:48:00.082313 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:00.082281    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56bc947947-gwfqd"
Sep 09 21:48:11.833820 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.833781    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"]
Sep 09 21:48:11.837595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.837572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.839950 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.839923    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b6wss\""
Sep 09 21:48:11.840076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.839926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:48:11.846056 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.846027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.846545 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.846520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.846752 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.846731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.846929 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.846912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.847097 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.847081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.847275 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.847258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtfx\" (UniqueName: \"kubernetes.io/projected/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-kube-api-access-rmtfx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.847445 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.847420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.847562 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.847502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.847562 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.847540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.848724 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.848698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"]
Sep 09 21:48:11.948815 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.948775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.948827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.948865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.948901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.948929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949211 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949211 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949211 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949211 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtfx\" (UniqueName: \"kubernetes.io/projected/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-kube-api-access-rmtfx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949408 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949460 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949669 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949810 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.949810 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.949681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.951447 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.951425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.951535 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.951460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.963740 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.963707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:11.963867 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:11.963833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtfx\" (UniqueName: \"kubernetes.io/projected/bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4-kube-api-access-rmtfx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9\" (UID: \"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:12.151360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:12.151322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:12.427791 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:12.427763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"]
Sep 09 21:48:12.430177 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:48:12.430143    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf48a25a_3a8a_4177_944f_e1ee5d9ef8c4.slice/crio-2dac5802680b55914f7567fe2924019eab29c47d14cd31bfb1c5a2c1d30e6ad3 WatchSource:0}: Error finding container 2dac5802680b55914f7567fe2924019eab29c47d14cd31bfb1c5a2c1d30e6ad3: Status 404 returned error can't find the container with id 2dac5802680b55914f7567fe2924019eab29c47d14cd31bfb1c5a2c1d30e6ad3
Sep 09 21:48:13.178738 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:13.178693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9" event={"ID":"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4","Type":"ContainerStarted","Data":"2dac5802680b55914f7567fe2924019eab29c47d14cd31bfb1c5a2c1d30e6ad3"}
Sep 09 21:48:17.112868 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:17.112820    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:48:17.113160 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:17.112921    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:48:17.113160 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:17.112952    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:48:17.201559 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:17.201514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9" event={"ID":"bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4","Type":"ContainerStarted","Data":"1a311d7e9a924b27fc6cfd863476a67ad9d2c822da004b2755171cbe5c705295"}
Sep 09 21:48:17.225634 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:17.225566    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9" podStartSLOduration=1.5450919330000001 podStartE2EDuration="6.225548428s" totalImagesPullingTime="4.680456495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:17.224011319 +0000 UTC m=+571.646374020" watchObservedRunningTime="2026-09-09 21:48:17.225548428 +0000 UTC m=+571.647911127"
Sep 09 21:48:18.151854 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:18.151816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:18.156767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:18.156738    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:18.205698 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:18.205664    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:18.207196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:18.207174    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9"
Sep 09 21:48:19.695743 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.695702    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"]
Sep 09 21:48:19.699229 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.699202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.702748 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.702720    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 21:48:19.702894 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.702819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-72vr8\""
Sep 09 21:48:19.702894 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.702838    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 21:48:19.703021 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.702750    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:48:19.703021 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.702740    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 21:48:19.703224 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.703199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 21:48:19.708374 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.708352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"]
Sep 09 21:48:19.826015 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.825964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a3b1a73d-66d5-4ada-8228-c4e682064f5f-manager-config\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.826198 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.826024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3b1a73d-66d5-4ada-8228-c4e682064f5f-metrics-cert\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.826198 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.826106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7sr6\" (UniqueName: \"kubernetes.io/projected/a3b1a73d-66d5-4ada-8228-c4e682064f5f-kube-api-access-g7sr6\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.826271 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.826194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3b1a73d-66d5-4ada-8228-c4e682064f5f-cert\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.927242 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.927207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3b1a73d-66d5-4ada-8228-c4e682064f5f-cert\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.927425 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.927268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a3b1a73d-66d5-4ada-8228-c4e682064f5f-manager-config\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.927425 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.927295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3b1a73d-66d5-4ada-8228-c4e682064f5f-metrics-cert\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.927425 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.927336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7sr6\" (UniqueName: \"kubernetes.io/projected/a3b1a73d-66d5-4ada-8228-c4e682064f5f-kube-api-access-g7sr6\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.928045 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.928017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a3b1a73d-66d5-4ada-8228-c4e682064f5f-manager-config\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.929913 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.929882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3b1a73d-66d5-4ada-8228-c4e682064f5f-cert\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.930008 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.929930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3b1a73d-66d5-4ada-8228-c4e682064f5f-metrics-cert\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:19.935792 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:19.935769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7sr6\" (UniqueName: \"kubernetes.io/projected/a3b1a73d-66d5-4ada-8228-c4e682064f5f-kube-api-access-g7sr6\") pod \"lws-controller-manager-798b8d4d4-cswmr\" (UID: \"a3b1a73d-66d5-4ada-8228-c4e682064f5f\") " pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:20.010127 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:20.010046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:20.287993 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:20.287910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"]
Sep 09 21:48:20.291037 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:48:20.291008    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b1a73d_66d5_4ada_8228_c4e682064f5f.slice/crio-926b0ed17382a1f70d3624419022ea4b02a43f818151c935e87b4ad92e663f9f WatchSource:0}: Error finding container 926b0ed17382a1f70d3624419022ea4b02a43f818151c935e87b4ad92e663f9f: Status 404 returned error can't find the container with id 926b0ed17382a1f70d3624419022ea4b02a43f818151c935e87b4ad92e663f9f
Sep 09 21:48:21.223369 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:21.223325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr" event={"ID":"a3b1a73d-66d5-4ada-8228-c4e682064f5f","Type":"ContainerStarted","Data":"926b0ed17382a1f70d3624419022ea4b02a43f818151c935e87b4ad92e663f9f"}
Sep 09 21:48:23.234631 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.234564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr" event={"ID":"a3b1a73d-66d5-4ada-8228-c4e682064f5f","Type":"ContainerStarted","Data":"109ec3100f6e300a1cdc90519815051deaa82d0a8818082ed64d0b5032899a94"}
Sep 09 21:48:23.235030 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.234672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:23.271890 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.271830    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr" podStartSLOduration=1.7825235940000002 podStartE2EDuration="4.271811133s" totalImagesPullingTime="2.489287539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:23.269888738 +0000 UTC m=+577.692251439" watchObservedRunningTime="2026-09-09 21:48:23.271811133 +0000 UTC m=+577.694173884"
Sep 09 21:48:23.949122 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.949087    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"]
Sep 09 21:48:23.957344 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.957309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:23.961010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.960984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 09 21:48:23.961010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.960984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tbbqd\""
Sep 09 21:48:23.969256 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.969226    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"]
Sep 09 21:48:23.973328 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.973309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:23.976197 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.976155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 21:48:23.976381 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.976360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gbxr8\""
Sep 09 21:48:23.978651 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.978625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"]
Sep 09 21:48:23.983226 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:23.983201    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"]
Sep 09 21:48:24.063535 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.063489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpj6w\" (UniqueName: \"kubernetes.io/projected/478bfb26-f142-4867-a37e-089756d3809b-kube-api-access-qpj6w\") pod \"llmisvc-controller-manager-777c6bd7d5-jmpp8\" (UID: \"478bfb26-f142-4867-a37e-089756d3809b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:24.063535 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.063536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb9f6ae7-1d80-4c29-899f-0f935069a4ab-cert\") pod \"kserve-controller-manager-d6f4b65b8-ndh64\" (UID: \"bb9f6ae7-1d80-4c29-899f-0f935069a4ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:24.063808 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.063593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnd7l\" (UniqueName: \"kubernetes.io/projected/bb9f6ae7-1d80-4c29-899f-0f935069a4ab-kube-api-access-bnd7l\") pod \"kserve-controller-manager-d6f4b65b8-ndh64\" (UID: \"bb9f6ae7-1d80-4c29-899f-0f935069a4ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:24.063808 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.063652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/478bfb26-f142-4867-a37e-089756d3809b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jmpp8\" (UID: \"478bfb26-f142-4867-a37e-089756d3809b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:24.169524 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.169423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpj6w\" (UniqueName: \"kubernetes.io/projected/478bfb26-f142-4867-a37e-089756d3809b-kube-api-access-qpj6w\") pod \"llmisvc-controller-manager-777c6bd7d5-jmpp8\" (UID: \"478bfb26-f142-4867-a37e-089756d3809b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:24.169524 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.169488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb9f6ae7-1d80-4c29-899f-0f935069a4ab-cert\") pod \"kserve-controller-manager-d6f4b65b8-ndh64\" (UID: \"bb9f6ae7-1d80-4c29-899f-0f935069a4ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:24.169795 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.169532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnd7l\" (UniqueName: \"kubernetes.io/projected/bb9f6ae7-1d80-4c29-899f-0f935069a4ab-kube-api-access-bnd7l\") pod \"kserve-controller-manager-d6f4b65b8-ndh64\" (UID: \"bb9f6ae7-1d80-4c29-899f-0f935069a4ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:24.169795 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.169564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/478bfb26-f142-4867-a37e-089756d3809b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jmpp8\" (UID: \"478bfb26-f142-4867-a37e-089756d3809b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:24.169795 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:48:24.169729    2579 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 09 21:48:24.169958 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:48:24.169821    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/478bfb26-f142-4867-a37e-089756d3809b-cert podName:478bfb26-f142-4867-a37e-089756d3809b nodeName:}" failed. No retries permitted until 2026-09-09 21:48:24.669786671 +0000 UTC m=+579.092149351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/478bfb26-f142-4867-a37e-089756d3809b-cert") pod "llmisvc-controller-manager-777c6bd7d5-jmpp8" (UID: "478bfb26-f142-4867-a37e-089756d3809b") : secret "llmisvc-webhook-server-cert" not found
Sep 09 21:48:24.172332 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.172298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb9f6ae7-1d80-4c29-899f-0f935069a4ab-cert\") pod \"kserve-controller-manager-d6f4b65b8-ndh64\" (UID: \"bb9f6ae7-1d80-4c29-899f-0f935069a4ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:24.178496 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.178471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpj6w\" (UniqueName: \"kubernetes.io/projected/478bfb26-f142-4867-a37e-089756d3809b-kube-api-access-qpj6w\") pod \"llmisvc-controller-manager-777c6bd7d5-jmpp8\" (UID: \"478bfb26-f142-4867-a37e-089756d3809b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:24.187003 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.186973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnd7l\" (UniqueName: \"kubernetes.io/projected/bb9f6ae7-1d80-4c29-899f-0f935069a4ab-kube-api-access-bnd7l\") pod \"kserve-controller-manager-d6f4b65b8-ndh64\" (UID: \"bb9f6ae7-1d80-4c29-899f-0f935069a4ab\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:24.268954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.268920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:24.542089 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.542063    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"]
Sep 09 21:48:24.544420 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:48:24.544385    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9f6ae7_1d80_4c29_899f_0f935069a4ab.slice/crio-98f3cb518f9d5947de1c9b83c719f163461e8a1cdab47253d97f364a9fdc0d87 WatchSource:0}: Error finding container 98f3cb518f9d5947de1c9b83c719f163461e8a1cdab47253d97f364a9fdc0d87: Status 404 returned error can't find the container with id 98f3cb518f9d5947de1c9b83c719f163461e8a1cdab47253d97f364a9fdc0d87
Sep 09 21:48:24.674303 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.674257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/478bfb26-f142-4867-a37e-089756d3809b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jmpp8\" (UID: \"478bfb26-f142-4867-a37e-089756d3809b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:24.676818 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.676792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/478bfb26-f142-4867-a37e-089756d3809b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jmpp8\" (UID: \"478bfb26-f142-4867-a37e-089756d3809b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:24.884756 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:24.884709    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:25.161401 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:25.161371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"]
Sep 09 21:48:25.163451 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:48:25.163420    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod478bfb26_f142_4867_a37e_089756d3809b.slice/crio-99529abd1e1aced76424d75fd807e6d1f9a53b5f62bf36d76c1e47cd0120d675 WatchSource:0}: Error finding container 99529abd1e1aced76424d75fd807e6d1f9a53b5f62bf36d76c1e47cd0120d675: Status 404 returned error can't find the container with id 99529abd1e1aced76424d75fd807e6d1f9a53b5f62bf36d76c1e47cd0120d675
Sep 09 21:48:25.244918 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:25.244882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64" event={"ID":"bb9f6ae7-1d80-4c29-899f-0f935069a4ab","Type":"ContainerStarted","Data":"98f3cb518f9d5947de1c9b83c719f163461e8a1cdab47253d97f364a9fdc0d87"}
Sep 09 21:48:25.246116 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:25.246088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8" event={"ID":"478bfb26-f142-4867-a37e-089756d3809b","Type":"ContainerStarted","Data":"99529abd1e1aced76424d75fd807e6d1f9a53b5f62bf36d76c1e47cd0120d675"}
Sep 09 21:48:28.271387 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:28.271295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64" event={"ID":"bb9f6ae7-1d80-4c29-899f-0f935069a4ab","Type":"ContainerStarted","Data":"9280a9673be029292d54201b6da146f8ec783fcca4219f434f717af9b0d25a0b"}
Sep 09 21:48:28.271789 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:28.271443    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:48:28.287456 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:28.287406    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64" podStartSLOduration=2.243706946 podStartE2EDuration="5.287387378s" totalImagesPullingTime="3.043680432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:28.286306076 +0000 UTC m=+582.708668787" watchObservedRunningTime="2026-09-09 21:48:28.287387378 +0000 UTC m=+582.709750238"
Sep 09 21:48:30.281729 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:30.281690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8" event={"ID":"478bfb26-f142-4867-a37e-089756d3809b","Type":"ContainerStarted","Data":"347868f0c051c3e65442e0b374131d86ebaea9a01ca80bb4b1bc8c5b47ee12e0"}
Sep 09 21:48:30.282159 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:30.281798    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:48:30.302278 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:30.302214    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8" podStartSLOduration=3.211220717 podStartE2EDuration="7.302191847s" totalImagesPullingTime="4.09097113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:30.299968037 +0000 UTC m=+584.722330738" watchObservedRunningTime="2026-09-09 21:48:30.302191847 +0000 UTC m=+584.724554549"
Sep 09 21:48:34.241504 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:34.241476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-798b8d4d4-cswmr"
Sep 09 21:48:59.281326 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:48:59.281291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ndh64"
Sep 09 21:49:01.288379 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:01.288347    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jmpp8"
Sep 09 21:49:38.447760 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.447719    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"]
Sep 09 21:49:38.451190 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.451172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:38.453836 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.453810    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 21:49:38.454788 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.454762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4rwt5\""
Sep 09 21:49:38.454898 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.454788    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 21:49:38.464536 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.464508    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"]
Sep 09 21:49:38.535724 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.535681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dg7d\" (UniqueName: \"kubernetes.io/projected/d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1-kube-api-access-6dg7d\") pod \"limitador-operator-controller-manager-74495c69f8-rjhtr\" (UID: \"d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:38.637043 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.637003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dg7d\" (UniqueName: \"kubernetes.io/projected/d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1-kube-api-access-6dg7d\") pod \"limitador-operator-controller-manager-74495c69f8-rjhtr\" (UID: \"d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:38.647735 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.647701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dg7d\" (UniqueName: \"kubernetes.io/projected/d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1-kube-api-access-6dg7d\") pod \"limitador-operator-controller-manager-74495c69f8-rjhtr\" (UID: \"d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:38.761547 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:38.761459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:39.036193 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:39.036162    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"]
Sep 09 21:49:39.039109 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:49:39.039073    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ac6ce1_b04a_4cda_bbfe_d0ee770e3da1.slice/crio-f3bdbd5c4ba170befedd22bddd60c51f3d0f3e1f52dced4763590c8175c2f728 WatchSource:0}: Error finding container f3bdbd5c4ba170befedd22bddd60c51f3d0f3e1f52dced4763590c8175c2f728: Status 404 returned error can't find the container with id f3bdbd5c4ba170befedd22bddd60c51f3d0f3e1f52dced4763590c8175c2f728
Sep 09 21:49:39.601099 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:39.601065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr" event={"ID":"d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1","Type":"ContainerStarted","Data":"f3bdbd5c4ba170befedd22bddd60c51f3d0f3e1f52dced4763590c8175c2f728"}
Sep 09 21:49:42.618323 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:42.617901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr" event={"ID":"d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1","Type":"ContainerStarted","Data":"8e0b40680a334df0fc27886a8307bc9c1f5b02cba7e3fe67d931de531cc75d1d"}
Sep 09 21:49:42.618823 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:42.618411    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:42.636074 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:42.636024    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr" podStartSLOduration=1.387062869 podStartE2EDuration="4.636008255s" totalImagesPullingTime="3.248945386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:42.634973922 +0000 UTC m=+657.057336623" watchObservedRunningTime="2026-09-09 21:49:42.636008255 +0000 UTC m=+657.058370954"
Sep 09 21:49:43.424470 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.424431    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"]
Sep 09 21:49:43.426852 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.426828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.429115 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.429080    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 21:49:43.429235 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.429178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 21:49:43.429305 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.429174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-252jf\""
Sep 09 21:49:43.437074 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.437054    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"]
Sep 09 21:49:43.480237 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.480210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.480358 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.480278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5hk\" (UniqueName: \"kubernetes.io/projected/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-kube-api-access-bs5hk\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.480358 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.480321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.580814 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.580778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5hk\" (UniqueName: \"kubernetes.io/projected/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-kube-api-access-bs5hk\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.580814 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.580821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.581009 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.580865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.581589 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.581567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.583354 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.583330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.589850 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.589827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5hk\" (UniqueName: \"kubernetes.io/projected/7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2-kube-api-access-bs5hk\") pod \"kuadrant-console-plugin-fbdd4f445-pnqnf\" (UID: \"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:43.736334 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:43.736240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"
Sep 09 21:49:44.003795 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:44.003770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf"]
Sep 09 21:49:44.006367 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:49:44.006339    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c6d4ad7_7888_4c91_8dc2_ecff50cdc8b2.slice/crio-bd69b0e075f7cd31e23af2e62d7512f359e5a13c84dcc5c0ee780b60dc7f33ce WatchSource:0}: Error finding container bd69b0e075f7cd31e23af2e62d7512f359e5a13c84dcc5c0ee780b60dc7f33ce: Status 404 returned error can't find the container with id bd69b0e075f7cd31e23af2e62d7512f359e5a13c84dcc5c0ee780b60dc7f33ce
Sep 09 21:49:44.628377 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:44.628342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf" event={"ID":"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2","Type":"ContainerStarted","Data":"bd69b0e075f7cd31e23af2e62d7512f359e5a13c84dcc5c0ee780b60dc7f33ce"}
Sep 09 21:49:49.654247 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:49.654204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf" event={"ID":"7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2","Type":"ContainerStarted","Data":"d3f9c672dadef9ed5784b497ee2eec8bb4074ed38cb832575878a9e6f8610f23"}
Sep 09 21:49:49.670126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:49.670071    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pnqnf" podStartSLOduration=1.8170289149999999 podStartE2EDuration="6.670054728s" totalImagesPullingTime="4.853025813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:49.668834967 +0000 UTC m=+664.091197666" watchObservedRunningTime="2026-09-09 21:49:49.670054728 +0000 UTC m=+664.092417428"
Sep 09 21:49:53.625028 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:53.624996    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:54.462373 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.462343    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"]
Sep 09 21:49:54.462587 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.462563    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" containerName="manager" containerID="cri-o://8e0b40680a334df0fc27886a8307bc9c1f5b02cba7e3fe67d931de531cc75d1d" gracePeriod=2
Sep 09 21:49:54.485826 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.485794    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"]
Sep 09 21:49:54.489416 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:49:54.489345    2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-rjhtr\": pods \"limitador-operator-controller-manager-74495c69f8-rjhtr\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:54.491713 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.491687    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"]
Sep 09 21:49:54.492073 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.492061    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" containerName="manager"
Sep 09 21:49:54.492117 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.492076    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" containerName="manager"
Sep 09 21:49:54.492151 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.492132    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" containerName="manager"
Sep 09 21:49:54.494170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.494150    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"
Sep 09 21:49:54.496371 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:49:54.496338    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rjhtr\" is forbidden: User \"system:node:ip-10-0-139-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-221.ec2.internal' and this object" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:54.507907 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.507882    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"]
Sep 09 21:49:54.586575 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.586536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cdn\" (UniqueName: \"kubernetes.io/projected/7167994c-bafe-4e71-81df-570f550e40a4-kube-api-access-58cdn\") pod \"limitador-operator-controller-manager-74495c69f8-sp7fz\" (UID: \"7167994c-bafe-4e71-81df-570f550e40a4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"
Sep 09 21:49:54.685327 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.685299    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" containerID="8e0b40680a334df0fc27886a8307bc9c1f5b02cba7e3fe67d931de531cc75d1d" exitCode=0
Sep 09 21:49:54.687774 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.687750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cdn\" (UniqueName: \"kubernetes.io/projected/7167994c-bafe-4e71-81df-570f550e40a4-kube-api-access-58cdn\") pod \"limitador-operator-controller-manager-74495c69f8-sp7fz\" (UID: \"7167994c-bafe-4e71-81df-570f550e40a4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"
Sep 09 21:49:54.699750 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.699729    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:54.700747 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.700725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cdn\" (UniqueName: \"kubernetes.io/projected/7167994c-bafe-4e71-81df-570f550e40a4-kube-api-access-58cdn\") pod \"limitador-operator-controller-manager-74495c69f8-sp7fz\" (UID: \"7167994c-bafe-4e71-81df-570f550e40a4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"
Sep 09 21:49:54.704414 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:49:54.704394    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rjhtr\" is forbidden: User \"system:node:ip-10-0-139-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-221.ec2.internal' and this object" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:54.788838 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.788745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dg7d\" (UniqueName: \"kubernetes.io/projected/d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1-kube-api-access-6dg7d\") pod \"d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1\" (UID: \"d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1\") "
Sep 09 21:49:54.790989 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.790953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1-kube-api-access-6dg7d" (OuterVolumeSpecName: "kube-api-access-6dg7d") pod "d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" (UID: "d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1"). InnerVolumeSpecName "kube-api-access-6dg7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:49:54.835163 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.835121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"
Sep 09 21:49:54.890323 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:54.890289    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dg7d\" (UniqueName: \"kubernetes.io/projected/d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1-kube-api-access-6dg7d\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:49:55.129453 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:55.129250    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"]
Sep 09 21:49:55.131723 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:49:55.131695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7167994c_bafe_4e71_81df_570f550e40a4.slice/crio-e68c08a2dfcb34b62ed1d2a3361ac6b02ec0650a7a18d92517f51485802c2fcb WatchSource:0}: Error finding container e68c08a2dfcb34b62ed1d2a3361ac6b02ec0650a7a18d92517f51485802c2fcb: Status 404 returned error can't find the container with id e68c08a2dfcb34b62ed1d2a3361ac6b02ec0650a7a18d92517f51485802c2fcb
Sep 09 21:49:55.691153 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:55.691123    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:55.691626 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:55.691170    2579 scope.go:117] "RemoveContainer" containerID="8e0b40680a334df0fc27886a8307bc9c1f5b02cba7e3fe67d931de531cc75d1d"
Sep 09 21:49:55.693268 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:55.693242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz" event={"ID":"7167994c-bafe-4e71-81df-570f550e40a4","Type":"ContainerStarted","Data":"72f97c988d81eafe9825bde83ce04e9158e5afab74c3edfeeecfcfff6b0f9369"}
Sep 09 21:49:55.693367 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:55.693292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz" event={"ID":"7167994c-bafe-4e71-81df-570f550e40a4","Type":"ContainerStarted","Data":"e68c08a2dfcb34b62ed1d2a3361ac6b02ec0650a7a18d92517f51485802c2fcb"}
Sep 09 21:49:55.693367 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:55.693352    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"
Sep 09 21:49:55.693460 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:49:55.693405    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rjhtr\" is forbidden: User \"system:node:ip-10-0-139-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-221.ec2.internal' and this object" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:55.711221 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:55.711143    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz" podStartSLOduration=1.711128019 podStartE2EDuration="1.711128019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:55.709615349 +0000 UTC m=+670.131978044" watchObservedRunningTime="2026-09-09 21:49:55.711128019 +0000 UTC m=+670.133490752"
Sep 09 21:49:55.711453 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:49:55.711425    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rjhtr\" is forbidden: User \"system:node:ip-10-0-139-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-221.ec2.internal' and this object" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:55.713057 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:49:55.713027    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rjhtr\" is forbidden: User \"system:node:ip-10-0-139-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-221.ec2.internal' and this object" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:56.090435 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:49:56.090333    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rjhtr\" is forbidden: User \"system:node:ip-10-0-139-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-221.ec2.internal' and this object" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rjhtr"
Sep 09 21:49:56.093861 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:49:56.093822    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1" path="/var/lib/kubelet/pods/d2ac6ce1-b04a-4cda-bbfe-d0ee770e3da1/volumes"
Sep 09 21:50:06.701954 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:06.701915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sp7fz"
Sep 09 21:50:22.671829 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.671791    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"]
Sep 09 21:50:22.692866 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.692828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"]
Sep 09 21:50:22.693031 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.692956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.695444 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.695423    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-647c7\""
Sep 09 21:50:22.740433 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.740433 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.740762 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.740762 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.740762 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.740762 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.740931 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.740931 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.740931 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.740857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnrd\" (UniqueName: \"kubernetes.io/projected/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-kube-api-access-lmnrd\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.841914 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.841866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.841914 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.841918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.841966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnrd\" (UniqueName: \"kubernetes.io/projected/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-kube-api-access-lmnrd\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842381 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842381 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842381 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842542 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842542 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842680 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.842680 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.843019 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.842996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.844184 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.844162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.844581 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.844558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.850695 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.850664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:22.850789 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:22.850736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnrd\" (UniqueName: \"kubernetes.io/projected/df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc-kube-api-access-lmnrd\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7vd78\" (UID: \"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:23.006053 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:23.005964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:23.279017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:23.278990    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"]
Sep 09 21:50:23.280900 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:50:23.280863    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf87cceb_f7da_4e26_bfcc_5c3fd4fcbfbc.slice/crio-d4a4788f0698842d82d0ca3be63cd4dcb336e76d46456c1610bc8991163da804 WatchSource:0}: Error finding container d4a4788f0698842d82d0ca3be63cd4dcb336e76d46456c1610bc8991163da804: Status 404 returned error can't find the container with id d4a4788f0698842d82d0ca3be63cd4dcb336e76d46456c1610bc8991163da804
Sep 09 21:50:23.283179 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:23.283143    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:50:23.283267 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:23.283228    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:50:23.283317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:23.283270    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:50:23.826776 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:23.826741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" event={"ID":"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc","Type":"ContainerStarted","Data":"72f28a179f37bca52f5cd15c82820eeb88d61a9c2c93e18df160401549734f90"}
Sep 09 21:50:23.827156 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:23.826796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" event={"ID":"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc","Type":"ContainerStarted","Data":"d4a4788f0698842d82d0ca3be63cd4dcb336e76d46456c1610bc8991163da804"}
Sep 09 21:50:23.846745 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:23.846695    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" podStartSLOduration=1.8466797019999999 podStartE2EDuration="1.846679702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:23.844337155 +0000 UTC m=+698.266699882" watchObservedRunningTime="2026-09-09 21:50:23.846679702 +0000 UTC m=+698.269042401"
Sep 09 21:50:24.006209 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:24.006164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:24.010915 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:24.010891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:24.833326 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:24.833299    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:24.834958 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:24.834933    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:50:25.562459 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:25.562424    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hlccb"]
Sep 09 21:50:25.569249 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:25.568037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb"
Sep 09 21:50:25.570800 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:25.570771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kf294\""
Sep 09 21:50:25.573039 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:25.573010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hlccb"]
Sep 09 21:50:25.671944 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:25.671903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxsr\" (UniqueName: \"kubernetes.io/projected/58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0-kube-api-access-7nxsr\") pod \"authorino-7bcfd8f9cd-hlccb\" (UID: \"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb"
Sep 09 21:50:25.772709 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:25.772677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxsr\" (UniqueName: \"kubernetes.io/projected/58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0-kube-api-access-7nxsr\") pod \"authorino-7bcfd8f9cd-hlccb\" (UID: \"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb"
Sep 09 21:50:25.780455 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:25.780426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxsr\" (UniqueName: \"kubernetes.io/projected/58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0-kube-api-access-7nxsr\") pod \"authorino-7bcfd8f9cd-hlccb\" (UID: \"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb"
Sep 09 21:50:25.880330 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:25.880292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb"
Sep 09 21:50:26.149958 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:26.149929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hlccb"]
Sep 09 21:50:26.152034 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:50:26.152005    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58bbc3d5_9aec_471c_8bd6_96a0aeadf5e0.slice/crio-aac84920f302c3825459614ccf956b4f96dc7693cc9e188c0a526f4f8b61f470 WatchSource:0}: Error finding container aac84920f302c3825459614ccf956b4f96dc7693cc9e188c0a526f4f8b61f470: Status 404 returned error can't find the container with id aac84920f302c3825459614ccf956b4f96dc7693cc9e188c0a526f4f8b61f470
Sep 09 21:50:26.842804 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:26.842771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb" event={"ID":"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0","Type":"ContainerStarted","Data":"aac84920f302c3825459614ccf956b4f96dc7693cc9e188c0a526f4f8b61f470"}
Sep 09 21:50:28.855213 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:28.855174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb" event={"ID":"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0","Type":"ContainerStarted","Data":"b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989"}
Sep 09 21:50:28.869597 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:28.869547    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb" podStartSLOduration=1.833691374 podStartE2EDuration="3.869532692s" totalImagesPullingTime="2.035841318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:28.868415862 +0000 UTC m=+703.290778565" watchObservedRunningTime="2026-09-09 21:50:28.869532692 +0000 UTC m=+703.291895392"
Sep 09 21:50:29.695057 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:29.695019    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hlccb"]
Sep 09 21:50:30.864210 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:30.864165    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb" podUID="58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0" containerName="authorino" containerID="cri-o://b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989" gracePeriod=30
Sep 09 21:50:31.123191 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.123132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb"
Sep 09 21:50:31.218948 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.218906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nxsr\" (UniqueName: \"kubernetes.io/projected/58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0-kube-api-access-7nxsr\") pod \"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0\" (UID: \"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0\") "
Sep 09 21:50:31.220899 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.220867    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0-kube-api-access-7nxsr" (OuterVolumeSpecName: "kube-api-access-7nxsr") pod "58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0" (UID: "58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0"). InnerVolumeSpecName "kube-api-access-7nxsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:50:31.320227 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.320197    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nxsr\" (UniqueName: \"kubernetes.io/projected/58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0-kube-api-access-7nxsr\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:50:31.869859 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.869821    2579 generic.go:356] "Generic (PLEG): container finished" podID="58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0" containerID="b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989" exitCode=0
Sep 09 21:50:31.870262 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.869865    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb"
Sep 09 21:50:31.870262 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.869905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb" event={"ID":"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0","Type":"ContainerDied","Data":"b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989"}
Sep 09 21:50:31.870262 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.869960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hlccb" event={"ID":"58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0","Type":"ContainerDied","Data":"aac84920f302c3825459614ccf956b4f96dc7693cc9e188c0a526f4f8b61f470"}
Sep 09 21:50:31.870262 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.869981    2579 scope.go:117] "RemoveContainer" containerID="b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989"
Sep 09 21:50:31.879309 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.879295    2579 scope.go:117] "RemoveContainer" containerID="b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989"
Sep 09 21:50:31.879572 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:50:31.879551    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989\": container with ID starting with b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989 not found: ID does not exist" containerID="b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989"
Sep 09 21:50:31.879694 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.879577    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989"} err="failed to get container status \"b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989\": rpc error: code = NotFound desc = could not find container \"b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989\": container with ID starting with b7a89c3cd318ed7f74b04aae600e7091593f13f71e81d1c6da56858fc058a989 not found: ID does not exist"
Sep 09 21:50:31.890233 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.890209    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hlccb"]
Sep 09 21:50:31.893501 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:31.893480    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hlccb"]
Sep 09 21:50:32.097080 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:32.092782    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0" path="/var/lib/kubelet/pods/58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0/volumes"
Sep 09 21:50:41.256660 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.256564    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"]
Sep 09 21:50:41.257168 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.257145    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0" containerName="authorino"
Sep 09 21:50:41.257218 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.257174    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0" containerName="authorino"
Sep 09 21:50:41.257320 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.257303    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bbc3d5-9aec-471c-8bd6-96a0aeadf5e0" containerName="authorino"
Sep 09 21:50:41.260270 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.260248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:41.269087 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.269058    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 21:50:41.269224 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.269116    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:50:41.270285 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.270260    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2vq5n\""
Sep 09 21:50:41.270403 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.270304    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:50:41.288329 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.288305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"]
Sep 09 21:50:41.315531 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.315505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc5257cb-82c7-48c1-9937-be1be1e2a18c-data\") pod \"postgres-69fb4594cf-kv6k9\" (UID: \"fc5257cb-82c7-48c1-9937-be1be1e2a18c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:41.315717 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.315695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92khm\" (UniqueName: \"kubernetes.io/projected/fc5257cb-82c7-48c1-9937-be1be1e2a18c-kube-api-access-92khm\") pod \"postgres-69fb4594cf-kv6k9\" (UID: \"fc5257cb-82c7-48c1-9937-be1be1e2a18c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:41.417194 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.417158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc5257cb-82c7-48c1-9937-be1be1e2a18c-data\") pod \"postgres-69fb4594cf-kv6k9\" (UID: \"fc5257cb-82c7-48c1-9937-be1be1e2a18c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:41.417406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.417212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92khm\" (UniqueName: \"kubernetes.io/projected/fc5257cb-82c7-48c1-9937-be1be1e2a18c-kube-api-access-92khm\") pod \"postgres-69fb4594cf-kv6k9\" (UID: \"fc5257cb-82c7-48c1-9937-be1be1e2a18c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:41.417641 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.417588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc5257cb-82c7-48c1-9937-be1be1e2a18c-data\") pod \"postgres-69fb4594cf-kv6k9\" (UID: \"fc5257cb-82c7-48c1-9937-be1be1e2a18c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:41.426758 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.426729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92khm\" (UniqueName: \"kubernetes.io/projected/fc5257cb-82c7-48c1-9937-be1be1e2a18c-kube-api-access-92khm\") pod \"postgres-69fb4594cf-kv6k9\" (UID: \"fc5257cb-82c7-48c1-9937-be1be1e2a18c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:41.571406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.571299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:41.841000 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.840974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"]
Sep 09 21:50:41.842847 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:50:41.842817    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5257cb_82c7_48c1_9937_be1be1e2a18c.slice/crio-042a44c7f488e95fd12a7f4499febcceabb51d1a83b5de47ad7ab043a955b62c WatchSource:0}: Error finding container 042a44c7f488e95fd12a7f4499febcceabb51d1a83b5de47ad7ab043a955b62c: Status 404 returned error can't find the container with id 042a44c7f488e95fd12a7f4499febcceabb51d1a83b5de47ad7ab043a955b62c
Sep 09 21:50:41.922480 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:41.922445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9" event={"ID":"fc5257cb-82c7-48c1-9937-be1be1e2a18c","Type":"ContainerStarted","Data":"042a44c7f488e95fd12a7f4499febcceabb51d1a83b5de47ad7ab043a955b62c"}
Sep 09 21:50:44.939678 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:44.939639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9" event={"ID":"fc5257cb-82c7-48c1-9937-be1be1e2a18c","Type":"ContainerStarted","Data":"b0ee2506b323bf5b0df4c5979d46d125cf99231c8bfa204f9f443398a729990c"}
Sep 09 21:50:44.940076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:44.939785    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:44.958333 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:44.958283    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9" podStartSLOduration=1.7743136769999999 podStartE2EDuration="3.958266165s" totalImagesPullingTime="2.183952488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:50:44.956000132 +0000 UTC m=+719.378362835" watchObservedRunningTime="2026-09-09 21:50:44.958266165 +0000 UTC m=+719.380628865"
Sep 09 21:50:50.973311 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:50.973278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv6k9"
Sep 09 21:50:51.459917 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.459878    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-9xmtv"]
Sep 09 21:50:51.466114 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.466089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-9xmtv"
Sep 09 21:50:51.468668 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.468616    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kf294\""
Sep 09 21:50:51.470309 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.470281    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-9xmtv"]
Sep 09 21:50:51.503414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.503374    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-9xmtv"]
Sep 09 21:50:51.503707 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:50:51.503687    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qvhml], unattached volumes=[], failed to process volumes=[kube-api-access-qvhml]: context canceled" pod="kuadrant-system/authorino-557f876f69-9xmtv" podUID="8b594a90-81bd-4552-8f1c-074d70c53e30"
Sep 09 21:50:51.530859 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.530820    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pm8hj"]
Sep 09 21:50:51.535184 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.535161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.537338 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.537315    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 21:50:51.542107 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.542075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pm8hj"]
Sep 09 21:50:51.610666 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.610621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhml\" (UniqueName: \"kubernetes.io/projected/8b594a90-81bd-4552-8f1c-074d70c53e30-kube-api-access-qvhml\") pod \"authorino-557f876f69-9xmtv\" (UID: \"8b594a90-81bd-4552-8f1c-074d70c53e30\") " pod="kuadrant-system/authorino-557f876f69-9xmtv"
Sep 09 21:50:51.610849 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.610707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ac3f1ff-7b50-4a66-b668-db5237650e04-tls-cert\") pod \"authorino-6d4dfc649f-pm8hj\" (UID: \"1ac3f1ff-7b50-4a66-b668-db5237650e04\") " pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.610849 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.610735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45wq9\" (UniqueName: \"kubernetes.io/projected/1ac3f1ff-7b50-4a66-b668-db5237650e04-kube-api-access-45wq9\") pod \"authorino-6d4dfc649f-pm8hj\" (UID: \"1ac3f1ff-7b50-4a66-b668-db5237650e04\") " pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.689976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.689939    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pm8hj"]
Sep 09 21:50:51.690275 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:50:51.690254    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-45wq9 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-pm8hj" podUID="1ac3f1ff-7b50-4a66-b668-db5237650e04"
Sep 09 21:50:51.711457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.711375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhml\" (UniqueName: \"kubernetes.io/projected/8b594a90-81bd-4552-8f1c-074d70c53e30-kube-api-access-qvhml\") pod \"authorino-557f876f69-9xmtv\" (UID: \"8b594a90-81bd-4552-8f1c-074d70c53e30\") " pod="kuadrant-system/authorino-557f876f69-9xmtv"
Sep 09 21:50:51.711457 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.711453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ac3f1ff-7b50-4a66-b668-db5237650e04-tls-cert\") pod \"authorino-6d4dfc649f-pm8hj\" (UID: \"1ac3f1ff-7b50-4a66-b668-db5237650e04\") " pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.711633 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.711486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45wq9\" (UniqueName: \"kubernetes.io/projected/1ac3f1ff-7b50-4a66-b668-db5237650e04-kube-api-access-45wq9\") pod \"authorino-6d4dfc649f-pm8hj\" (UID: \"1ac3f1ff-7b50-4a66-b668-db5237650e04\") " pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.714071 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.714037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ac3f1ff-7b50-4a66-b668-db5237650e04-tls-cert\") pod \"authorino-6d4dfc649f-pm8hj\" (UID: \"1ac3f1ff-7b50-4a66-b668-db5237650e04\") " pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.723422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.723398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhml\" (UniqueName: \"kubernetes.io/projected/8b594a90-81bd-4552-8f1c-074d70c53e30-kube-api-access-qvhml\") pod \"authorino-557f876f69-9xmtv\" (UID: \"8b594a90-81bd-4552-8f1c-074d70c53e30\") " pod="kuadrant-system/authorino-557f876f69-9xmtv"
Sep 09 21:50:51.723546 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.723440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45wq9\" (UniqueName: \"kubernetes.io/projected/1ac3f1ff-7b50-4a66-b668-db5237650e04-kube-api-access-45wq9\") pod \"authorino-6d4dfc649f-pm8hj\" (UID: \"1ac3f1ff-7b50-4a66-b668-db5237650e04\") " pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.973743 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.973652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.974172 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.973652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-9xmtv"
Sep 09 21:50:51.982679 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.982652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:51.985620 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:51.985581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-9xmtv"
Sep 09 21:50:52.114993 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.114953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45wq9\" (UniqueName: \"kubernetes.io/projected/1ac3f1ff-7b50-4a66-b668-db5237650e04-kube-api-access-45wq9\") pod \"1ac3f1ff-7b50-4a66-b668-db5237650e04\" (UID: \"1ac3f1ff-7b50-4a66-b668-db5237650e04\") "
Sep 09 21:50:52.115167 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.115016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvhml\" (UniqueName: \"kubernetes.io/projected/8b594a90-81bd-4552-8f1c-074d70c53e30-kube-api-access-qvhml\") pod \"8b594a90-81bd-4552-8f1c-074d70c53e30\" (UID: \"8b594a90-81bd-4552-8f1c-074d70c53e30\") "
Sep 09 21:50:52.115167 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.115042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ac3f1ff-7b50-4a66-b668-db5237650e04-tls-cert\") pod \"1ac3f1ff-7b50-4a66-b668-db5237650e04\" (UID: \"1ac3f1ff-7b50-4a66-b668-db5237650e04\") "
Sep 09 21:50:52.117275 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.117234    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b594a90-81bd-4552-8f1c-074d70c53e30-kube-api-access-qvhml" (OuterVolumeSpecName: "kube-api-access-qvhml") pod "8b594a90-81bd-4552-8f1c-074d70c53e30" (UID: "8b594a90-81bd-4552-8f1c-074d70c53e30"). InnerVolumeSpecName "kube-api-access-qvhml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:50:52.117275 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.117257    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac3f1ff-7b50-4a66-b668-db5237650e04-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1ac3f1ff-7b50-4a66-b668-db5237650e04" (UID: "1ac3f1ff-7b50-4a66-b668-db5237650e04"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:50:52.117275 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.117242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac3f1ff-7b50-4a66-b668-db5237650e04-kube-api-access-45wq9" (OuterVolumeSpecName: "kube-api-access-45wq9") pod "1ac3f1ff-7b50-4a66-b668-db5237650e04" (UID: "1ac3f1ff-7b50-4a66-b668-db5237650e04"). InnerVolumeSpecName "kube-api-access-45wq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:50:52.215821 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.215781    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45wq9\" (UniqueName: \"kubernetes.io/projected/1ac3f1ff-7b50-4a66-b668-db5237650e04-kube-api-access-45wq9\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:50:52.215821 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.215812    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhml\" (UniqueName: \"kubernetes.io/projected/8b594a90-81bd-4552-8f1c-074d70c53e30-kube-api-access-qvhml\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:50:52.215821 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.215822    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ac3f1ff-7b50-4a66-b668-db5237650e04-tls-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:50:52.978506 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.978468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pm8hj"
Sep 09 21:50:52.979018 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:52.978468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-9xmtv"
Sep 09 21:50:53.008293 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:53.008254    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-9xmtv"]
Sep 09 21:50:53.009956 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:53.009930    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-9xmtv"]
Sep 09 21:50:53.029941 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:53.029901    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pm8hj"]
Sep 09 21:50:53.032960 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:53.032932    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pm8hj"]
Sep 09 21:50:54.091656 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:54.091598    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac3f1ff-7b50-4a66-b668-db5237650e04" path="/var/lib/kubelet/pods/1ac3f1ff-7b50-4a66-b668-db5237650e04/volumes"
Sep 09 21:50:54.092173 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:50:54.091968    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b594a90-81bd-4552-8f1c-074d70c53e30" path="/var/lib/kubelet/pods/8b594a90-81bd-4552-8f1c-074d70c53e30/volumes"
Sep 09 21:51:37.110531 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.110498    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"]
Sep 09 21:51:37.113151 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.113123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:37.116024 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.115995    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 21:51:37.116163 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.116142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7hwm4\""
Sep 09 21:51:37.126886 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.126865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"]
Sep 09 21:51:37.228418 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.228364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9gfh\" (UniqueName: \"kubernetes.io/projected/6e710c3b-e23c-4789-ba15-0088d6d84359-kube-api-access-b9gfh\") pod \"payload-pre-processing-59fbc47bf8-6jbpm\" (UID: \"6e710c3b-e23c-4789-ba15-0088d6d84359\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:37.228633 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.228429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e710c3b-e23c-4789-ba15-0088d6d84359-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6jbpm\" (UID: \"6e710c3b-e23c-4789-ba15-0088d6d84359\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:37.329687 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.329640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9gfh\" (UniqueName: \"kubernetes.io/projected/6e710c3b-e23c-4789-ba15-0088d6d84359-kube-api-access-b9gfh\") pod \"payload-pre-processing-59fbc47bf8-6jbpm\" (UID: \"6e710c3b-e23c-4789-ba15-0088d6d84359\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:37.329687 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.329689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e710c3b-e23c-4789-ba15-0088d6d84359-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6jbpm\" (UID: \"6e710c3b-e23c-4789-ba15-0088d6d84359\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:37.330475 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.330453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e710c3b-e23c-4789-ba15-0088d6d84359-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6jbpm\" (UID: \"6e710c3b-e23c-4789-ba15-0088d6d84359\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:37.337771 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.337747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9gfh\" (UniqueName: \"kubernetes.io/projected/6e710c3b-e23c-4789-ba15-0088d6d84359-kube-api-access-b9gfh\") pod \"payload-pre-processing-59fbc47bf8-6jbpm\" (UID: \"6e710c3b-e23c-4789-ba15-0088d6d84359\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:37.423637 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.423513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:37.691286 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:37.691258    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"]
Sep 09 21:51:37.693982 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:51:37.693950    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e710c3b_e23c_4789_ba15_0088d6d84359.slice/crio-32f6a16cad44251e0f8b138369f4cbb75de0a806d85cfc87dae56c2b9ff7f1be WatchSource:0}: Error finding container 32f6a16cad44251e0f8b138369f4cbb75de0a806d85cfc87dae56c2b9ff7f1be: Status 404 returned error can't find the container with id 32f6a16cad44251e0f8b138369f4cbb75de0a806d85cfc87dae56c2b9ff7f1be
Sep 09 21:51:38.190863 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:38.190827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm" event={"ID":"6e710c3b-e23c-4789-ba15-0088d6d84359","Type":"ContainerStarted","Data":"32f6a16cad44251e0f8b138369f4cbb75de0a806d85cfc87dae56c2b9ff7f1be"}
Sep 09 21:51:41.227313 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:41.227204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm" event={"ID":"6e710c3b-e23c-4789-ba15-0088d6d84359","Type":"ContainerStarted","Data":"2f5a434d320200e09516acf3a4aa5f8af3f54c11c3773478c05526ea5867e3a7"}
Sep 09 21:51:41.227313 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:41.227271    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:41.244012 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:41.243945    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm" podStartSLOduration=1.379508409 podStartE2EDuration="4.243926678s" totalImagesPullingTime="2.864418269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:51:41.24245234 +0000 UTC m=+775.664815042" watchObservedRunningTime="2026-09-09 21:51:41.243926678 +0000 UTC m=+775.666289377"
Sep 09 21:51:47.259984 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:47.259941    2579 generic.go:356] "Generic (PLEG): container finished" podID="df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc" containerID="72f28a179f37bca52f5cd15c82820eeb88d61a9c2c93e18df160401549734f90" exitCode=0
Sep 09 21:51:47.260455 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:47.259997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" event={"ID":"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc","Type":"ContainerDied","Data":"72f28a179f37bca52f5cd15c82820eeb88d61a9c2c93e18df160401549734f90"}
Sep 09 21:51:47.260504 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:47.260457    2579 scope.go:117] "RemoveContainer" containerID="72f28a179f37bca52f5cd15c82820eeb88d61a9c2c93e18df160401549734f90"
Sep 09 21:51:47.261494 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:47.261465    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:51:47.261554 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:47.261535    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:51:47.261589 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:47.261567    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:51:48.266869 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:48.266835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" event={"ID":"df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc","Type":"ContainerStarted","Data":"da402cb3eacd15dff09193d58d6d13813eb8781c647bb53acd1cf01ec12f0bbf"}
Sep 09 21:51:49.007056 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:49.007017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:51:49.007429 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:49.007407    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7vd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 21:51:49.007594 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:49.007570    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" podUID="df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 21:51:50.007156 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:50.007115    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7vd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 21:51:50.007640 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:50.007186    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" podUID="df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 21:51:51.006429 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:51.006389    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7vd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 21:51:51.006755 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:51.006719    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" podUID="df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 21:51:52.007690 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:52.007213    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7vd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 21:51:52.007690 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:52.007283    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" podUID="df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 21:51:52.235470 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:52.235435    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6jbpm"
Sep 09 21:51:53.006507 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:53.006468    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7vd78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 21:51:53.006835 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:53.006801    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78" podUID="df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 21:51:54.010907 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:54.010877    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:51:54.011358 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:54.011143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:51:54.011824 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:51:54.011805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7vd78"
Sep 09 21:52:21.065294 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.065203    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"]
Sep 09 21:52:21.067524 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.067508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.070187 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.070157    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5x6f2\""
Sep 09 21:52:21.070187 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.070180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 21:52:21.071023 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.071004    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 21:52:21.071162 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.071136    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 21:52:21.080332 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.080309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"]
Sep 09 21:52:21.166281 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.166244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.166281 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.166282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.166493 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.166308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.166493 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.166334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.166493 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.166367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.166493 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.166462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8pj\" (UniqueName: \"kubernetes.io/projected/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-kube-api-access-zs8pj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.267493 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.267442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.267745 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.267513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.267745 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.267560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.267745 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.267670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8pj\" (UniqueName: \"kubernetes.io/projected/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-kube-api-access-zs8pj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.267745 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.267727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.267961 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.267754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.268026 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.267952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.268026 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.267976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.268134 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.268105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.270001 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.269971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.270193 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.270173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.275901 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.275878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8pj\" (UniqueName: \"kubernetes.io/projected/f7d7b5f1-7797-4122-8d11-2021c1d3db3e-kube-api-access-zs8pj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv\" (UID: \"f7d7b5f1-7797-4122-8d11-2021c1d3db3e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.378486 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.378448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:21.649592 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:21.649567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"]
Sep 09 21:52:21.651419 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:52:21.651391    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d7b5f1_7797_4122_8d11_2021c1d3db3e.slice/crio-84586a3e76a5b3606aeeab2e67a7ebdca947e2dfdb1440da6001293ce11d95f4 WatchSource:0}: Error finding container 84586a3e76a5b3606aeeab2e67a7ebdca947e2dfdb1440da6001293ce11d95f4: Status 404 returned error can't find the container with id 84586a3e76a5b3606aeeab2e67a7ebdca947e2dfdb1440da6001293ce11d95f4
Sep 09 21:52:22.439828 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:22.439789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv" event={"ID":"f7d7b5f1-7797-4122-8d11-2021c1d3db3e","Type":"ContainerStarted","Data":"84586a3e76a5b3606aeeab2e67a7ebdca947e2dfdb1440da6001293ce11d95f4"}
Sep 09 21:52:23.446352 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:23.446316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv" event={"ID":"f7d7b5f1-7797-4122-8d11-2021c1d3db3e","Type":"ContainerStarted","Data":"83ae46ae7d42f9a7d8a64100a52e074d4f97937574bddbbff717e343ac0b364b"}
Sep 09 21:52:23.465105 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:23.465053    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv" podStartSLOduration=1.470565893 podStartE2EDuration="2.465037646s" totalImagesPullingTime="994.471753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:23.463262039 +0000 UTC m=+817.885624767" watchObservedRunningTime="2026-09-09 21:52:23.465037646 +0000 UTC m=+817.887400349"
Sep 09 21:52:31.379623 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:31.379554    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:31.380154 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:31.379647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:31.391988 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:31.391965    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:31.499096 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:31.499068    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv"
Sep 09 21:52:37.355814 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.355778    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"]
Sep 09 21:52:37.359417 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.359392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.362009 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.361979    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 21:52:37.368933 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.368910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"]
Sep 09 21:52:37.416451 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.416419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.416649 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.416489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.416649 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.416512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.416649 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.416548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8647\" (UniqueName: \"kubernetes.io/projected/b250c38e-6f70-464b-9d57-2289baf2d3e9-kube-api-access-h8647\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.416776 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.416644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.416776 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.416680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b250c38e-6f70-464b-9d57-2289baf2d3e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.517714 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.517679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.517714 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.517715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.517964 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.517766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8647\" (UniqueName: \"kubernetes.io/projected/b250c38e-6f70-464b-9d57-2289baf2d3e9-kube-api-access-h8647\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.517964 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.517804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.517964 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.517831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b250c38e-6f70-464b-9d57-2289baf2d3e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.517964 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.517893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.518251 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.518133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.518251 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.518175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.518357 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.518312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.520048 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.520017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b250c38e-6f70-464b-9d57-2289baf2d3e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.520333 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.520314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b250c38e-6f70-464b-9d57-2289baf2d3e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.525579 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.525554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8647\" (UniqueName: \"kubernetes.io/projected/b250c38e-6f70-464b-9d57-2289baf2d3e9-kube-api-access-h8647\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4994x\" (UID: \"b250c38e-6f70-464b-9d57-2289baf2d3e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.670425 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.670335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:37.943230 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:37.943153    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"]
Sep 09 21:52:37.946722 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:52:37.946693    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb250c38e_6f70_464b_9d57_2289baf2d3e9.slice/crio-8351488d0495b7c5469102b9ccc53f7e1ba3a0dad3c6d221e45279dee517f51e WatchSource:0}: Error finding container 8351488d0495b7c5469102b9ccc53f7e1ba3a0dad3c6d221e45279dee517f51e: Status 404 returned error can't find the container with id 8351488d0495b7c5469102b9ccc53f7e1ba3a0dad3c6d221e45279dee517f51e
Sep 09 21:52:38.522865 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:38.522768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x" event={"ID":"b250c38e-6f70-464b-9d57-2289baf2d3e9","Type":"ContainerStarted","Data":"3802670ee3043234f08f492f52fa450ca36ad15b174de84c0389b1978165fcef"}
Sep 09 21:52:38.523315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:38.522884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x" event={"ID":"b250c38e-6f70-464b-9d57-2289baf2d3e9","Type":"ContainerStarted","Data":"8351488d0495b7c5469102b9ccc53f7e1ba3a0dad3c6d221e45279dee517f51e"}
Sep 09 21:52:38.542209 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:38.542150    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x" podStartSLOduration=1.355407761 podStartE2EDuration="1.542134819s" totalImagesPullingTime="186.727058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:38.540150589 +0000 UTC m=+832.962513290" watchObservedRunningTime="2026-09-09 21:52:38.542134819 +0000 UTC m=+832.964497558"
Sep 09 21:52:47.671473 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:47.671434    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:47.671898 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:47.671519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:47.684366 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:47.684335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:52:48.585461 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:52:48.585434    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4994x"
Sep 09 21:54:41.592873 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.592835    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"]
Sep 09 21:54:41.597183 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.597157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.599669 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.599645    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5aaa80-openshift-default-dockercfg-pbgkf\""
Sep 09 21:54:41.610564 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.610529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"]
Sep 09 21:54:41.698843 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.698803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-socket\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.699033 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.698858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133a7cdf-2548-4f74-979c-633d884a16ef-istio-podinfo\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.699033 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.699022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-envoy\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.699157 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.699062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-istio-token\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.699157 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.699087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-data\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.699268 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.699181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-certs\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.699268 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.699220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133a7cdf-2548-4f74-979c-633d884a16ef-istiod-ca-cert\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.699380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.699267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4k6z\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-kube-api-access-m4k6z\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.699380 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.699311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-credential-socket\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800164 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-envoy\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800164 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-istio-token\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-data\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-certs\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133a7cdf-2548-4f74-979c-633d884a16ef-istiod-ca-cert\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4k6z\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-kube-api-access-m4k6z\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-credential-socket\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-socket\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133a7cdf-2548-4f74-979c-633d884a16ef-istio-podinfo\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800831 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-data\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800831 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-certs\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.800953 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.800930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-credential-socket\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.801044 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.801021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-socket\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.801309 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.801290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133a7cdf-2548-4f74-979c-633d884a16ef-istiod-ca-cert\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.802905 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.802885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-envoy\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.803353 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.803333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133a7cdf-2548-4f74-979c-633d884a16ef-istio-podinfo\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.808214 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.808188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-istio-token\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.809000 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.808979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4k6z\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-kube-api-access-m4k6z\") pod \"e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") " pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:41.910229 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:41.910193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:42.188846 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:42.188813    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"]
Sep 09 21:54:42.191212 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:54:42.191181    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133a7cdf_2548_4f74_979c_633d884a16ef.slice/crio-11fe2cfc052c70438b98899b46096088c0bffc032f111dbbacd2a0f8a487964b WatchSource:0}: Error finding container 11fe2cfc052c70438b98899b46096088c0bffc032f111dbbacd2a0f8a487964b: Status 404 returned error can't find the container with id 11fe2cfc052c70438b98899b46096088c0bffc032f111dbbacd2a0f8a487964b
Sep 09 21:54:42.193911 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:42.193879    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:54:42.193978 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:42.193949    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:54:42.194028 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:42.193975    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:54:43.137812 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:43.137770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" event={"ID":"133a7cdf-2548-4f74-979c-633d884a16ef","Type":"ContainerStarted","Data":"668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390"}
Sep 09 21:54:43.138315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:43.137849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" event={"ID":"133a7cdf-2548-4f74-979c-633d884a16ef","Type":"ContainerStarted","Data":"11fe2cfc052c70438b98899b46096088c0bffc032f111dbbacd2a0f8a487964b"}
Sep 09 21:54:43.160750 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:43.160695    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podStartSLOduration=2.160677329 podStartE2EDuration="2.160677329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:43.159586208 +0000 UTC m=+957.581948909" watchObservedRunningTime="2026-09-09 21:54:43.160677329 +0000 UTC m=+957.583040028"
Sep 09 21:54:43.910779 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:43.910735    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:43.915330 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:43.915304    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:44.142974 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:44.142934    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:44.143861 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:44.143839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:54:46.937114 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:46.937075    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"]
Sep 09 21:54:46.940959 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:46.940935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:46.943451 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:46.943420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5aaa80\""
Sep 09 21:54:46.943616 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:46.943581    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5aaa80-dockercfg-756vj\""
Sep 09 21:54:46.944785 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:46.944758    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"]
Sep 09 21:54:46.948371 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:46.948344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:46.951537 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:46.951510    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"]
Sep 09 21:54:46.958435 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:46.958401    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"]
Sep 09 21:54:47.058521 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.058481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c26949-7864-4fe7-a707-4fcbf0707078-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6\" (UID: \"11c26949-7864-4fe7-a707-4fcbf0707078\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:47.058521 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.058520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdf9\" (UniqueName: \"kubernetes.io/projected/11c26949-7864-4fe7-a707-4fcbf0707078-kube-api-access-5rdf9\") pod \"payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6\" (UID: \"11c26949-7864-4fe7-a707-4fcbf0707078\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:47.058808 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.058569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktzj\" (UniqueName: \"kubernetes.io/projected/c0cf131e-eded-47d8-9f02-02d12d171a91-kube-api-access-7ktzj\") pod \"payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk\" (UID: \"c0cf131e-eded-47d8-9f02-02d12d171a91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:47.058808 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.058658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cf131e-eded-47d8-9f02-02d12d171a91-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk\" (UID: \"c0cf131e-eded-47d8-9f02-02d12d171a91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:47.159429 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.159392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c26949-7864-4fe7-a707-4fcbf0707078-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6\" (UID: \"11c26949-7864-4fe7-a707-4fcbf0707078\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:47.159429 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.159428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdf9\" (UniqueName: \"kubernetes.io/projected/11c26949-7864-4fe7-a707-4fcbf0707078-kube-api-access-5rdf9\") pod \"payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6\" (UID: \"11c26949-7864-4fe7-a707-4fcbf0707078\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:47.159672 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.159460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktzj\" (UniqueName: \"kubernetes.io/projected/c0cf131e-eded-47d8-9f02-02d12d171a91-kube-api-access-7ktzj\") pod \"payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk\" (UID: \"c0cf131e-eded-47d8-9f02-02d12d171a91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:47.159672 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.159495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cf131e-eded-47d8-9f02-02d12d171a91-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk\" (UID: \"c0cf131e-eded-47d8-9f02-02d12d171a91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:47.160399 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.160372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cf131e-eded-47d8-9f02-02d12d171a91-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk\" (UID: \"c0cf131e-eded-47d8-9f02-02d12d171a91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:47.160441 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.160372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c26949-7864-4fe7-a707-4fcbf0707078-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6\" (UID: \"11c26949-7864-4fe7-a707-4fcbf0707078\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:47.175848 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.175816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdf9\" (UniqueName: \"kubernetes.io/projected/11c26949-7864-4fe7-a707-4fcbf0707078-kube-api-access-5rdf9\") pod \"payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6\" (UID: \"11c26949-7864-4fe7-a707-4fcbf0707078\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:47.175990 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.175819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktzj\" (UniqueName: \"kubernetes.io/projected/c0cf131e-eded-47d8-9f02-02d12d171a91-kube-api-access-7ktzj\") pod \"payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk\" (UID: \"c0cf131e-eded-47d8-9f02-02d12d171a91\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:47.254713 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.254625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:47.265171 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.264944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:47.535256 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.535227    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"]
Sep 09 21:54:47.537422 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:54:47.537381    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c26949_7864_4fe7_a707_4fcbf0707078.slice/crio-5b9032d8c49a3d23c99104bd1c7dd58bfad41f46cdb79b4b486abf864eb393bf WatchSource:0}: Error finding container 5b9032d8c49a3d23c99104bd1c7dd58bfad41f46cdb79b4b486abf864eb393bf: Status 404 returned error can't find the container with id 5b9032d8c49a3d23c99104bd1c7dd58bfad41f46cdb79b4b486abf864eb393bf
Sep 09 21:54:47.546090 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:47.546065    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"]
Sep 09 21:54:47.547949 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:54:47.547917    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cf131e_eded_47d8_9f02_02d12d171a91.slice/crio-372b947ab9f968132cb9e1eec365ca7e6534a300fc519657bf7e6203678e4660 WatchSource:0}: Error finding container 372b947ab9f968132cb9e1eec365ca7e6534a300fc519657bf7e6203678e4660: Status 404 returned error can't find the container with id 372b947ab9f968132cb9e1eec365ca7e6534a300fc519657bf7e6203678e4660
Sep 09 21:54:48.166647 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:48.166586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6" event={"ID":"11c26949-7864-4fe7-a707-4fcbf0707078","Type":"ContainerStarted","Data":"0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a"}
Sep 09 21:54:48.167088 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:48.166670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6" event={"ID":"11c26949-7864-4fe7-a707-4fcbf0707078","Type":"ContainerStarted","Data":"5b9032d8c49a3d23c99104bd1c7dd58bfad41f46cdb79b4b486abf864eb393bf"}
Sep 09 21:54:48.167088 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:48.166806    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:48.168492 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:48.168467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk" event={"ID":"c0cf131e-eded-47d8-9f02-02d12d171a91","Type":"ContainerStarted","Data":"3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef"}
Sep 09 21:54:48.168627 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:48.168509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk" event={"ID":"c0cf131e-eded-47d8-9f02-02d12d171a91","Type":"ContainerStarted","Data":"372b947ab9f968132cb9e1eec365ca7e6534a300fc519657bf7e6203678e4660"}
Sep 09 21:54:48.168627 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:48.168536    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:48.191749 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:48.191702    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6" podStartSLOduration=2.191686428 podStartE2EDuration="2.191686428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:48.190463887 +0000 UTC m=+962.612826587" watchObservedRunningTime="2026-09-09 21:54:48.191686428 +0000 UTC m=+962.614049184"
Sep 09 21:54:48.209375 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:48.209320    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk" podStartSLOduration=2.209304339 podStartE2EDuration="2.209304339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:48.206529506 +0000 UTC m=+962.628892208" watchObservedRunningTime="2026-09-09 21:54:48.209304339 +0000 UTC m=+962.631667039"
Sep 09 21:54:49.621445 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.621404    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"]
Sep 09 21:54:49.625489 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.625466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.627897 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.627870    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ff5f88f5-openshift-default-dockercfg-9vvzn\""
Sep 09 21:54:49.640175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.640146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"]
Sep 09 21:54:49.685928 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.685888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-podinfo\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.686136 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.685949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-envoy\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.686136 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.685987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-credential-socket\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.686136 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.686034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-socket\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.686136 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.686090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-certs\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.686136 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.686131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbm5\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-kube-api-access-4tbm5\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.686360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.686210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istiod-ca-cert\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.686360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.686276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-data\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.686360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.686317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-token\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787067 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-certs\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787067 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbm5\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-kube-api-access-4tbm5\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787325 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istiod-ca-cert\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787325 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-data\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787325 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-token\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787325 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-podinfo\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787325 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-envoy\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787325 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-credential-socket\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787703 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-socket\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787703 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-certs\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787703 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-socket\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.787940 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.787922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-credential-socket\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.788198 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.788175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-data\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.788375 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.788354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istiod-ca-cert\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.789884 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.789860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-envoy\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.790249 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.790222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-podinfo\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.796567 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.796543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-token\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.796829 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.796806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbm5\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-kube-api-access-4tbm5\") pod \"e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") " pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:49.936270 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:49.936176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:50.214268 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:50.214242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"]
Sep 09 21:54:50.215877 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:54:50.215847    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b12a913_e5ed_4eba_b56f_05f17254e8d1.slice/crio-322b58d2b06de95c393eb568aecedb536418ace3ce26073079259eda296a3891 WatchSource:0}: Error finding container 322b58d2b06de95c393eb568aecedb536418ace3ce26073079259eda296a3891: Status 404 returned error can't find the container with id 322b58d2b06de95c393eb568aecedb536418ace3ce26073079259eda296a3891
Sep 09 21:54:50.218230 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:50.218199    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:54:50.218310 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:50.218264    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:54:50.218310 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:50.218294    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:54:51.188345 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:51.188305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" event={"ID":"0b12a913-e5ed-4eba-b56f-05f17254e8d1","Type":"ContainerStarted","Data":"08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65"}
Sep 09 21:54:51.188768 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:51.188364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" event={"ID":"0b12a913-e5ed-4eba-b56f-05f17254e8d1","Type":"ContainerStarted","Data":"322b58d2b06de95c393eb568aecedb536418ace3ce26073079259eda296a3891"}
Sep 09 21:54:51.211554 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:51.211504    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" podStartSLOduration=2.211489182 podStartE2EDuration="2.211489182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:51.20829757 +0000 UTC m=+965.630660271" watchObservedRunningTime="2026-09-09 21:54:51.211489182 +0000 UTC m=+965.633851882"
Sep 09 21:54:51.937092 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:51.937062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:51.942063 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:51.942034    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:52.193054 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:52.192973    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:52.194155 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:52.194134    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:54:54.577026 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.576990    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"]
Sep 09 21:54:54.580707 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.580691    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:54.583518 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.583496    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ff5f88f5-dockercfg-sd7n5\""
Sep 09 21:54:54.583673 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.583651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ff5f88f5\""
Sep 09 21:54:54.588471 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.588407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"]
Sep 09 21:54:54.614628 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.599360    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"]
Sep 09 21:54:54.630863 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.630830    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"]
Sep 09 21:54:54.631153 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.631136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:54.744234 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.744193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppphj\" (UniqueName: \"kubernetes.io/projected/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-kube-api-access-ppphj\") pod \"payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc\" (UID: \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\") " pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:54.744461 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.744261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a06bc03-54ab-40c5-9336-67a23ff19cf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv\" (UID: \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:54.744461 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.744305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxr5\" (UniqueName: \"kubernetes.io/projected/4a06bc03-54ab-40c5-9336-67a23ff19cf0-kube-api-access-4lxr5\") pod \"payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv\" (UID: \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:54.744575 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.744456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-plugins-config-volume\") pod \"payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc\" (UID: \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\") " pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:54.845324 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.845228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a06bc03-54ab-40c5-9336-67a23ff19cf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv\" (UID: \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:54.845324 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.845284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxr5\" (UniqueName: \"kubernetes.io/projected/4a06bc03-54ab-40c5-9336-67a23ff19cf0-kube-api-access-4lxr5\") pod \"payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv\" (UID: \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:54.845566 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.845353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-plugins-config-volume\") pod \"payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc\" (UID: \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\") " pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:54.845566 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.845383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppphj\" (UniqueName: \"kubernetes.io/projected/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-kube-api-access-ppphj\") pod \"payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc\" (UID: \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\") " pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:54.846092 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.846064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a06bc03-54ab-40c5-9336-67a23ff19cf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv\" (UID: \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:54.846193 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.846162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-plugins-config-volume\") pod \"payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc\" (UID: \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\") " pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:54.853246 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.853222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppphj\" (UniqueName: \"kubernetes.io/projected/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-kube-api-access-ppphj\") pod \"payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc\" (UID: \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\") " pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:54.853459 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.853435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxr5\" (UniqueName: \"kubernetes.io/projected/4a06bc03-54ab-40c5-9336-67a23ff19cf0-kube-api-access-4lxr5\") pod \"payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv\" (UID: \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:54.893268 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.893235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:54.943836 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:54.943802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:55.193164 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:55.193135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"]
Sep 09 21:54:55.194444 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:54:55.194419    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a06bc03_54ab_40c5_9336_67a23ff19cf0.slice/crio-357c1c6884b95717510ec39da1725fa6513483111c5034e94f9daca8741b3137 WatchSource:0}: Error finding container 357c1c6884b95717510ec39da1725fa6513483111c5034e94f9daca8741b3137: Status 404 returned error can't find the container with id 357c1c6884b95717510ec39da1725fa6513483111c5034e94f9daca8741b3137
Sep 09 21:54:55.231973 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:55.231820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv" event={"ID":"4a06bc03-54ab-40c5-9336-67a23ff19cf0","Type":"ContainerStarted","Data":"357c1c6884b95717510ec39da1725fa6513483111c5034e94f9daca8741b3137"}
Sep 09 21:54:55.256471 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:55.256410    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"]
Sep 09 21:54:55.259179 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:54:55.259133    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6982e9c7_2c2e_4615_8a72_c3d0c0c1b700.slice/crio-ee16dc614a56d27861220ad5eae2eddf665a9f3d9aa066af6ca714f65a29185d WatchSource:0}: Error finding container ee16dc614a56d27861220ad5eae2eddf665a9f3d9aa066af6ca714f65a29185d: Status 404 returned error can't find the container with id ee16dc614a56d27861220ad5eae2eddf665a9f3d9aa066af6ca714f65a29185d
Sep 09 21:54:56.240533 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:56.240465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv" event={"ID":"4a06bc03-54ab-40c5-9336-67a23ff19cf0","Type":"ContainerStarted","Data":"c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90"}
Sep 09 21:54:56.241068 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:56.241031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:54:56.242698 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:56.242668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc" event={"ID":"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700","Type":"ContainerStarted","Data":"837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65"}
Sep 09 21:54:56.242816 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:56.242716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc" event={"ID":"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700","Type":"ContainerStarted","Data":"ee16dc614a56d27861220ad5eae2eddf665a9f3d9aa066af6ca714f65a29185d"}
Sep 09 21:54:56.242816 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:56.242747    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:54:56.268966 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:56.268918    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv" podStartSLOduration=2.268897586 podStartE2EDuration="2.268897586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:56.265914503 +0000 UTC m=+970.688277203" watchObservedRunningTime="2026-09-09 21:54:56.268897586 +0000 UTC m=+970.691260288"
Sep 09 21:54:58.187280 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.187215    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc" podStartSLOduration=4.187197123 podStartE2EDuration="4.187197123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:56.291548011 +0000 UTC m=+970.713910711" watchObservedRunningTime="2026-09-09 21:54:58.187197123 +0000 UTC m=+972.609559823"
Sep 09 21:54:58.188207 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.188176    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"]
Sep 09 21:54:58.192170 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.192148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:58.198036 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.198005    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-594ab9\""
Sep 09 21:54:58.223072 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.223034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"]
Sep 09 21:54:58.288833 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.288796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-plugins-config-volume\") pod \"payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8\" (UID: \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\") " pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:58.289020 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.288971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghp5v\" (UniqueName: \"kubernetes.io/projected/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-kube-api-access-ghp5v\") pod \"payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8\" (UID: \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\") " pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:58.389615 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.389573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghp5v\" (UniqueName: \"kubernetes.io/projected/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-kube-api-access-ghp5v\") pod \"payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8\" (UID: \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\") " pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:58.389795 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.389675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-plugins-config-volume\") pod \"payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8\" (UID: \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\") " pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:58.390389 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.390367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-plugins-config-volume\") pod \"payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8\" (UID: \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\") " pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:58.397952 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.397916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghp5v\" (UniqueName: \"kubernetes.io/projected/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-kube-api-access-ghp5v\") pod \"payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8\" (UID: \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\") " pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:58.504451 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.504354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:58.787747 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:58.787665    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"]
Sep 09 21:54:59.175780 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:59.175746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:54:59.175983 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:59.175836    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:54:59.268112 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:59.268073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8" event={"ID":"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c","Type":"ContainerStarted","Data":"dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c"}
Sep 09 21:54:59.268631 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:59.268141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8" event={"ID":"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c","Type":"ContainerStarted","Data":"0d9219fdbc26e49f080fde18c008058f9583c35328213403cd702f76a62db280"}
Sep 09 21:54:59.268631 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:59.268186    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:54:59.286948 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:54:59.286871    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8" podStartSLOduration=1.286846876 podStartE2EDuration="1.286846876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:54:59.284807789 +0000 UTC m=+973.707170490" watchObservedRunningTime="2026-09-09 21:54:59.286846876 +0000 UTC m=+973.709209577"
Sep 09 21:55:00.184792 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:00.184754    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:55:00.184962 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:00.184820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:55:01.281828 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:01.281796    2579 generic.go:356] "Generic (PLEG): container finished" podID="133a7cdf-2548-4f74-979c-633d884a16ef" containerID="668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390" exitCode=0
Sep 09 21:55:01.282297 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:01.281867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" event={"ID":"133a7cdf-2548-4f74-979c-633d884a16ef","Type":"ContainerDied","Data":"668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390"}
Sep 09 21:55:01.282344 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:01.282333    2579 scope.go:117] "RemoveContainer" containerID="668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390"
Sep 09 21:55:01.283475 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:01.283450    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:55:01.283534 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:01.283521    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:55:01.283571 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:01.283556    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:55:02.291515 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:02.291481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" event={"ID":"133a7cdf-2548-4f74-979c-633d884a16ef","Type":"ContainerStarted","Data":"95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c"}
Sep 09 21:55:02.911364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:02.911318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:55:02.911587 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:02.911504    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:55:02.911587 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:02.911549    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:55:03.910770 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:03.910729    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:55:03.911345 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:03.911314    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:55:04.910753 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:04.910716    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:55:04.911045 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:04.911012    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:55:05.911565 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:05.911513    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:55:05.912125 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:05.911589    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:55:06.910796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:06.910746    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:55:06.911014 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:06.910844    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:55:07.252230 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:07.252145    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:55:07.252715 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:07.252336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:55:07.910757 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:07.910708    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 09 21:55:07.911066 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:07.911028    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 09 21:55:08.234901 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:08.234773    2579 patch_prober.go:28] interesting pod/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:55:08.234901 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:08.234847    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:55:08.915040 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:08.915005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:55:08.915505 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:08.915374    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:55:08.916119 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:08.916096    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:55:09.363117 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:09.363087    2579 generic.go:356] "Generic (PLEG): container finished" podID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerID="08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65" exitCode=0
Sep 09 21:55:09.363297 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:09.363162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" event={"ID":"0b12a913-e5ed-4eba-b56f-05f17254e8d1","Type":"ContainerDied","Data":"08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65"}
Sep 09 21:55:09.363760 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:09.363742    2579 scope.go:117] "RemoveContainer" containerID="08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65"
Sep 09 21:55:09.364818 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:09.364795    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:55:09.364893 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:09.364858    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:55:09.364893 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:09.364888    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:55:10.276042 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:10.276008    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:55:10.371174 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:10.371140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" event={"ID":"0b12a913-e5ed-4eba-b56f-05f17254e8d1","Type":"ContainerStarted","Data":"56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954"}
Sep 09 21:55:10.936557 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:10.936519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:55:10.936781 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:10.936757    2579 patch_prober.go:28] interesting pod/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:55:10.936829 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:10.936808    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:55:11.937419 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:11.937383    2579 patch_prober.go:28] interesting pod/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:55:11.937868 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:11.937440    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:55:12.937330 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:12.937293    2579 patch_prober.go:28] interesting pod/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:55:12.937510 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:12.937350    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:55:13.937429 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:13.937390    2579 patch_prober.go:28] interesting pod/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:55:13.937647 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:13.937455    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:55:14.937563 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:14.937521    2579 patch_prober.go:28] interesting pod/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 21:55:14.937782 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:14.937588    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 21:55:15.940195 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:15.940162    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:55:15.940585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:15.940461    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:55:15.941262 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:15.941245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:55:17.988002 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:17.987956    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"]
Sep 09 21:55:18.000049 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.000002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"
Sep 09 21:55:18.005482 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.005455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 21:55:18.007402 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.007369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"]
Sep 09 21:55:18.110987 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.110942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ab2a2a-82f1-412e-877f-3d94862b9aaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx\" (UID: \"02ab2a2a-82f1-412e-877f-3d94862b9aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"
Sep 09 21:55:18.212488 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.212441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ab2a2a-82f1-412e-877f-3d94862b9aaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx\" (UID: \"02ab2a2a-82f1-412e-877f-3d94862b9aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"
Sep 09 21:55:18.213332 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.213305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ab2a2a-82f1-412e-877f-3d94862b9aaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx\" (UID: \"02ab2a2a-82f1-412e-877f-3d94862b9aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"
Sep 09 21:55:18.319246 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.319133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"
Sep 09 21:55:18.584824 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.584745    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"]
Sep 09 21:55:18.587809 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.587788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:18.589907 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.589888    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ebe9b0\""
Sep 09 21:55:18.590032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.589896    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ebe9b0-dockercfg-2nghk\""
Sep 09 21:55:18.598234 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.598203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"]
Sep 09 21:55:18.604571 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.604546    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"]
Sep 09 21:55:18.607824 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:55:18.607800    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ab2a2a_82f1_412e_877f_3d94862b9aaa.slice/crio-2e4353b4c8303de630f9fca5927c52b75813423dc6e178347ceab10b0b68ea1e WatchSource:0}: Error finding container 2e4353b4c8303de630f9fca5927c52b75813423dc6e178347ceab10b0b68ea1e: Status 404 returned error can't find the container with id 2e4353b4c8303de630f9fca5927c52b75813423dc6e178347ceab10b0b68ea1e
Sep 09 21:55:18.610353 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.610326    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:55:18.617017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.616991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a12e563-fb9c-4fdf-9bef-ee851285bb73-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d\" (UID: \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:18.617099 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.617046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv962\" (UniqueName: \"kubernetes.io/projected/8a12e563-fb9c-4fdf-9bef-ee851285bb73-kube-api-access-rv962\") pod \"payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d\" (UID: \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:18.718531 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.718496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv962\" (UniqueName: \"kubernetes.io/projected/8a12e563-fb9c-4fdf-9bef-ee851285bb73-kube-api-access-rv962\") pod \"payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d\" (UID: \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:18.718766 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.718664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a12e563-fb9c-4fdf-9bef-ee851285bb73-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d\" (UID: \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:18.719364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.719343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a12e563-fb9c-4fdf-9bef-ee851285bb73-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d\" (UID: \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:18.727634 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.727591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv962\" (UniqueName: \"kubernetes.io/projected/8a12e563-fb9c-4fdf-9bef-ee851285bb73-kube-api-access-rv962\") pod \"payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d\" (UID: \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:18.900405 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:18.900364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:19.172915 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:19.172890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"]
Sep 09 21:55:19.174340 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:55:19.174314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a12e563_fb9c_4fdf_9bef_ee851285bb73.slice/crio-2ec657c732857d27a98abd9e76c63402989a51085008204a3f333fb9f76a8dee WatchSource:0}: Error finding container 2ec657c732857d27a98abd9e76c63402989a51085008204a3f333fb9f76a8dee: Status 404 returned error can't find the container with id 2ec657c732857d27a98abd9e76c63402989a51085008204a3f333fb9f76a8dee
Sep 09 21:55:19.432422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:19.432320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx" event={"ID":"02ab2a2a-82f1-412e-877f-3d94862b9aaa","Type":"ContainerStarted","Data":"2e4353b4c8303de630f9fca5927c52b75813423dc6e178347ceab10b0b68ea1e"}
Sep 09 21:55:19.434139 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:19.434112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d" event={"ID":"8a12e563-fb9c-4fdf-9bef-ee851285bb73","Type":"ContainerStarted","Data":"1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826"}
Sep 09 21:55:19.434289 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:19.434155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d" event={"ID":"8a12e563-fb9c-4fdf-9bef-ee851285bb73","Type":"ContainerStarted","Data":"2ec657c732857d27a98abd9e76c63402989a51085008204a3f333fb9f76a8dee"}
Sep 09 21:55:19.434289 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:19.434197    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:19.451571 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:19.451517    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d" podStartSLOduration=1.451499015 podStartE2EDuration="1.451499015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:19.449471843 +0000 UTC m=+993.871834556" watchObservedRunningTime="2026-09-09 21:55:19.451499015 +0000 UTC m=+993.873861715"
Sep 09 21:55:21.447209 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:21.447174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx" event={"ID":"02ab2a2a-82f1-412e-877f-3d94862b9aaa","Type":"ContainerStarted","Data":"06ff7a4eb846d6ce0f3cadbcedc8e1718807a0934b17c778b1da7ca949569fee"}
Sep 09 21:55:21.465095 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:21.465042    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx" podStartSLOduration=2.06974576 podStartE2EDuration="4.465025518s" totalImagesPullingTime="2.395279758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:21.46330552 +0000 UTC m=+995.885668222" watchObservedRunningTime="2026-09-09 21:55:21.465025518 +0000 UTC m=+995.887388219"
Sep 09 21:55:22.454210 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:22.454170    2579 generic.go:356] "Generic (PLEG): container finished" podID="02ab2a2a-82f1-412e-877f-3d94862b9aaa" containerID="06ff7a4eb846d6ce0f3cadbcedc8e1718807a0934b17c778b1da7ca949569fee" exitCode=0
Sep 09 21:55:22.454683 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:22.454255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx" event={"ID":"02ab2a2a-82f1-412e-877f-3d94862b9aaa","Type":"ContainerDied","Data":"06ff7a4eb846d6ce0f3cadbcedc8e1718807a0934b17c778b1da7ca949569fee"}
Sep 09 21:55:23.604859 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:23.604830    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"
Sep 09 21:55:23.672794 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:23.672757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ab2a2a-82f1-412e-877f-3d94862b9aaa-maas-api-service-ca\") pod \"02ab2a2a-82f1-412e-877f-3d94862b9aaa\" (UID: \"02ab2a2a-82f1-412e-877f-3d94862b9aaa\") "
Sep 09 21:55:23.673168 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:23.673140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ab2a2a-82f1-412e-877f-3d94862b9aaa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02ab2a2a-82f1-412e-877f-3d94862b9aaa" (UID: "02ab2a2a-82f1-412e-877f-3d94862b9aaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:23.774407 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:23.774305    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ab2a2a-82f1-412e-877f-3d94862b9aaa-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:24.467353 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:24.467268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx" event={"ID":"02ab2a2a-82f1-412e-877f-3d94862b9aaa","Type":"ContainerDied","Data":"2e4353b4c8303de630f9fca5927c52b75813423dc6e178347ceab10b0b68ea1e"}
Sep 09 21:55:24.467353 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:24.467331    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e4353b4c8303de630f9fca5927c52b75813423dc6e178347ceab10b0b68ea1e"
Sep 09 21:55:24.467353 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:24.467286    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"
Sep 09 21:55:30.443147 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:30.443109    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:55:34.845023 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:34.844982    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"]
Sep 09 21:55:34.845583 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:34.845248    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc" podUID="6982e9c7-2c2e-4615-8a72-c3d0c0c1b700" containerName="payload-processing" containerID="cri-o://837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65" gracePeriod=30
Sep 09 21:55:34.852415 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:34.852382    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"]
Sep 09 21:55:34.852649 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:34.852627    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv" podUID="4a06bc03-54ab-40c5-9336-67a23ff19cf0" containerName="payload-pre-processing" containerID="cri-o://c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90" gracePeriod=30
Sep 09 21:55:35.020681 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.020653    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"]
Sep 09 21:55:35.025367 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.025310    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff5f88f5-xf6sx"]
Sep 09 21:55:35.109497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.109093    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:55:35.142967 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.142932    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"]
Sep 09 21:55:35.143225 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.143195    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy" containerID="cri-o://56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954" gracePeriod=30
Sep 09 21:55:35.168006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.167951    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:55:35.200054 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.199553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-plugins-config-volume\") pod \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\" (UID: \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\") "
Sep 09 21:55:35.200054 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.199748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppphj\" (UniqueName: \"kubernetes.io/projected/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-kube-api-access-ppphj\") pod \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\" (UID: \"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700\") "
Sep 09 21:55:35.200054 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.199981    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6982e9c7-2c2e-4615-8a72-c3d0c0c1b700" (UID: "6982e9c7-2c2e-4615-8a72-c3d0c0c1b700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:35.200651 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.200626    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:35.202757 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.202729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-kube-api-access-ppphj" (OuterVolumeSpecName: "kube-api-access-ppphj") pod "6982e9c7-2c2e-4615-8a72-c3d0c0c1b700" (UID: "6982e9c7-2c2e-4615-8a72-c3d0c0c1b700"). InnerVolumeSpecName "kube-api-access-ppphj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:35.301019 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.300981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a06bc03-54ab-40c5-9336-67a23ff19cf0-plugins-config-volume\") pod \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\" (UID: \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\") "
Sep 09 21:55:35.301019 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.301028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lxr5\" (UniqueName: \"kubernetes.io/projected/4a06bc03-54ab-40c5-9336-67a23ff19cf0-kube-api-access-4lxr5\") pod \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\" (UID: \"4a06bc03-54ab-40c5-9336-67a23ff19cf0\") "
Sep 09 21:55:35.301675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.301394    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a06bc03-54ab-40c5-9336-67a23ff19cf0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a06bc03-54ab-40c5-9336-67a23ff19cf0" (UID: "4a06bc03-54ab-40c5-9336-67a23ff19cf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:35.301675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.301537    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppphj\" (UniqueName: \"kubernetes.io/projected/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700-kube-api-access-ppphj\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:35.301675 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.301556    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a06bc03-54ab-40c5-9336-67a23ff19cf0-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:35.303689 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.303659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a06bc03-54ab-40c5-9336-67a23ff19cf0-kube-api-access-4lxr5" (OuterVolumeSpecName: "kube-api-access-4lxr5") pod "4a06bc03-54ab-40c5-9336-67a23ff19cf0" (UID: "4a06bc03-54ab-40c5-9336-67a23ff19cf0"). InnerVolumeSpecName "kube-api-access-4lxr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:35.402526 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.402436    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lxr5\" (UniqueName: \"kubernetes.io/projected/4a06bc03-54ab-40c5-9336-67a23ff19cf0-kube-api-access-4lxr5\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:35.529788 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.529755    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a06bc03-54ab-40c5-9336-67a23ff19cf0" containerID="c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90" exitCode=0
Sep 09 21:55:35.529979 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.529808    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"
Sep 09 21:55:35.529979 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.529847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv" event={"ID":"4a06bc03-54ab-40c5-9336-67a23ff19cf0","Type":"ContainerDied","Data":"c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90"}
Sep 09 21:55:35.529979 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.529901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv" event={"ID":"4a06bc03-54ab-40c5-9336-67a23ff19cf0","Type":"ContainerDied","Data":"357c1c6884b95717510ec39da1725fa6513483111c5034e94f9daca8741b3137"}
Sep 09 21:55:35.529979 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.529930    2579 scope.go:117] "RemoveContainer" containerID="c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90"
Sep 09 21:55:35.531314 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.531282    2579 generic.go:356] "Generic (PLEG): container finished" podID="6982e9c7-2c2e-4615-8a72-c3d0c0c1b700" containerID="837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65" exitCode=0
Sep 09 21:55:35.531444 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.531315    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"
Sep 09 21:55:35.531444 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.531369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc" event={"ID":"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700","Type":"ContainerDied","Data":"837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65"}
Sep 09 21:55:35.531557 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.531465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc" event={"ID":"6982e9c7-2c2e-4615-8a72-c3d0c0c1b700","Type":"ContainerDied","Data":"ee16dc614a56d27861220ad5eae2eddf665a9f3d9aa066af6ca714f65a29185d"}
Sep 09 21:55:35.541283 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.541257    2579 scope.go:117] "RemoveContainer" containerID="c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90"
Sep 09 21:55:35.541730 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:55:35.541695    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90\": container with ID starting with c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90 not found: ID does not exist" containerID="c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90"
Sep 09 21:55:35.541844 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.541740    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90"} err="failed to get container status \"c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90\": rpc error: code = NotFound desc = could not find container \"c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90\": container with ID starting with c5dba5122068b5533c38f497a958ada50380aeb08e01feaa7d2ad4428bd1da90 not found: ID does not exist"
Sep 09 21:55:35.541844 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.541763    2579 scope.go:117] "RemoveContainer" containerID="837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65"
Sep 09 21:55:35.551126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.551105    2579 scope.go:117] "RemoveContainer" containerID="837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65"
Sep 09 21:55:35.551433 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:55:35.551415    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65\": container with ID starting with 837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65 not found: ID does not exist" containerID="837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65"
Sep 09 21:55:35.551534 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.551440    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65"} err="failed to get container status \"837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65\": rpc error: code = NotFound desc = could not find container \"837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65\": container with ID starting with 837d4507d88c8acc7b598b1a03e013448028ab3479d1bf3dd7350fb55d5bae65 not found: ID does not exist"
Sep 09 21:55:35.553040 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.553013    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"]
Sep 09 21:55:35.557780 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.557753    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ff5f88f5-56dfd64754-mjtlv"]
Sep 09 21:55:35.569344 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.569310    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"]
Sep 09 21:55:35.584192 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:35.584158    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ff5f88f5-7bf9498755-r6sgc"]
Sep 09 21:55:36.093229 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:36.093190    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02ab2a2a-82f1-412e-877f-3d94862b9aaa" path="/var/lib/kubelet/pods/02ab2a2a-82f1-412e-877f-3d94862b9aaa/volumes"
Sep 09 21:55:36.093592 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:36.093570    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a06bc03-54ab-40c5-9336-67a23ff19cf0" path="/var/lib/kubelet/pods/4a06bc03-54ab-40c5-9336-67a23ff19cf0/volumes"
Sep 09 21:55:36.093920 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:36.093906    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6982e9c7-2c2e-4615-8a72-c3d0c0c1b700" path="/var/lib/kubelet/pods/6982e9c7-2c2e-4615-8a72-c3d0c0c1b700/volumes"
Sep 09 21:55:39.230786 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.230752    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"]
Sep 09 21:55:39.231341 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231321    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6982e9c7-2c2e-4615-8a72-c3d0c0c1b700" containerName="payload-processing"
Sep 09 21:55:39.231445 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231344    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6982e9c7-2c2e-4615-8a72-c3d0c0c1b700" containerName="payload-processing"
Sep 09 21:55:39.231445 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231360    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ab2a2a-82f1-412e-877f-3d94862b9aaa" containerName="revoke-keys"
Sep 09 21:55:39.231445 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231367    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ab2a2a-82f1-412e-877f-3d94862b9aaa" containerName="revoke-keys"
Sep 09 21:55:39.231445 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231386    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a06bc03-54ab-40c5-9336-67a23ff19cf0" containerName="payload-pre-processing"
Sep 09 21:55:39.231445 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231394    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a06bc03-54ab-40c5-9336-67a23ff19cf0" containerName="payload-pre-processing"
Sep 09 21:55:39.231656 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231490    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a06bc03-54ab-40c5-9336-67a23ff19cf0" containerName="payload-pre-processing"
Sep 09 21:55:39.231656 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231500    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6982e9c7-2c2e-4615-8a72-c3d0c0c1b700" containerName="payload-processing"
Sep 09 21:55:39.231656 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.231512    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ab2a2a-82f1-412e-877f-3d94862b9aaa" containerName="revoke-keys"
Sep 09 21:55:39.234636 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.234619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:39.237389 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.237368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9e97e70d\""
Sep 09 21:55:39.237497 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.237368    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9e97e70d-dockercfg-qh555\""
Sep 09 21:55:39.245998 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.245964    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"]
Sep 09 21:55:39.346089 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.346026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-kube-api-access-znlgn\") pod \"payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f\" (UID: \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:39.346289 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.346239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f\" (UID: \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:39.447413 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.447354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-kube-api-access-znlgn\") pod \"payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f\" (UID: \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:39.447657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.447464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f\" (UID: \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:39.448167 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.448140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f\" (UID: \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:39.455183 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.455160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-kube-api-access-znlgn\") pod \"payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f\" (UID: \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:39.546193 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.546104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:39.824667 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:39.824638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"]
Sep 09 21:55:39.826707 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:55:39.826671    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea64c48f_f42e_4284_95bf_0a3a2cf94f2c.slice/crio-6da7e381bba1485078b5e1e89f14344822cc8a96f22234a052cb9a9144321f05 WatchSource:0}: Error finding container 6da7e381bba1485078b5e1e89f14344822cc8a96f22234a052cb9a9144321f05: Status 404 returned error can't find the container with id 6da7e381bba1485078b5e1e89f14344822cc8a96f22234a052cb9a9144321f05
Sep 09 21:55:40.390048 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.390015    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:55:40.454980 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.454892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istiod-ca-cert\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.454980 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.454933    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-envoy\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.454980 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.454949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-data\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.454980 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.454975    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-socket\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.455318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455023    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-podinfo\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.455318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455082    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-token\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.455318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455111    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-credential-socket\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.455318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455182    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-certs\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.455318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455223    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tbm5\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-kube-api-access-4tbm5\") pod \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\" (UID: \"0b12a913-e5ed-4eba-b56f-05f17254e8d1\") "
Sep 09 21:55:40.455318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455282    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:55:40.455667 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455399    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:40.455667 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:40.455667 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455533    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:40.455667 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455653    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.455905 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455673    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.455905 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455686    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.455905 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455700    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.455905 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.455772    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:40.457730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.457704    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:55:40.457730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.457703    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:55:40.457730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.457727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:40.457907 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.457763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-kube-api-access-4tbm5" (OuterVolumeSpecName: "kube-api-access-4tbm5") pod "0b12a913-e5ed-4eba-b56f-05f17254e8d1" (UID: "0b12a913-e5ed-4eba-b56f-05f17254e8d1"). InnerVolumeSpecName "kube-api-access-4tbm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:55:40.556657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.556597    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.556657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.556648    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.556657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.556657    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.556657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.556665    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.556923 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.556675    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbm5\" (UniqueName: \"kubernetes.io/projected/0b12a913-e5ed-4eba-b56f-05f17254e8d1-kube-api-access-4tbm5\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:55:40.561317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.561290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f" event={"ID":"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c","Type":"ContainerStarted","Data":"385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa"}
Sep 09 21:55:40.561446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.561352    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:55:40.561446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.561377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f" event={"ID":"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c","Type":"ContainerStarted","Data":"6da7e381bba1485078b5e1e89f14344822cc8a96f22234a052cb9a9144321f05"}
Sep 09 21:55:40.563046 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.563025    2579 generic.go:356] "Generic (PLEG): container finished" podID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerID="56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954" exitCode=0
Sep 09 21:55:40.563141 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.563059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" event={"ID":"0b12a913-e5ed-4eba-b56f-05f17254e8d1","Type":"ContainerDied","Data":"56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954"}
Sep 09 21:55:40.563141 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.563062    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"
Sep 09 21:55:40.563141 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.563101    2579 scope.go:117] "RemoveContainer" containerID="56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954"
Sep 09 21:55:40.563247 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.563093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25" event={"ID":"0b12a913-e5ed-4eba-b56f-05f17254e8d1","Type":"ContainerDied","Data":"322b58d2b06de95c393eb568aecedb536418ace3ce26073079259eda296a3891"}
Sep 09 21:55:40.572643 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.572622    2579 scope.go:117] "RemoveContainer" containerID="08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65"
Sep 09 21:55:40.578117 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.578075    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f" podStartSLOduration=1.5780618880000001 podStartE2EDuration="1.578061888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:55:40.576341533 +0000 UTC m=+1014.998704234" watchObservedRunningTime="2026-09-09 21:55:40.578061888 +0000 UTC m=+1015.000424629"
Sep 09 21:55:40.581825 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.581806    2579 scope.go:117] "RemoveContainer" containerID="56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954"
Sep 09 21:55:40.582145 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:55:40.582124    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954\": container with ID starting with 56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954 not found: ID does not exist" containerID="56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954"
Sep 09 21:55:40.582218 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.582154    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954"} err="failed to get container status \"56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954\": rpc error: code = NotFound desc = could not find container \"56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954\": container with ID starting with 56d842cb3497eb56d706f1061f9f042168e80a693809bd407ec79a3d17838954 not found: ID does not exist"
Sep 09 21:55:40.582218 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.582176    2579 scope.go:117] "RemoveContainer" containerID="08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65"
Sep 09 21:55:40.582542 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:55:40.582518    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65\": container with ID starting with 08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65 not found: ID does not exist" containerID="08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65"
Sep 09 21:55:40.582624 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.582546    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65"} err="failed to get container status \"08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65\": rpc error: code = NotFound desc = could not find container \"08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65\": container with ID starting with 08745fcfc7dd193886d86a16fcfba12765f574f490c8204ef9f09e8cd2619f65 not found: ID does not exist"
Sep 09 21:55:40.589461 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.589436    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"]
Sep 09 21:55:40.592824 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:40.592803    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ff5f88f5-openshift-default-5b665c7776-5fg25"]
Sep 09 21:55:42.092114 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:42.092080    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" path="/var/lib/kubelet/pods/0b12a913-e5ed-4eba-b56f-05f17254e8d1/volumes"
Sep 09 21:55:51.572547 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:55:51.572511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:56:03.320857 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.320820    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"]
Sep 09 21:56:03.321269 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.321255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy"
Sep 09 21:56:03.321269 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.321270    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy"
Sep 09 21:56:03.321356 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.321281    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy"
Sep 09 21:56:03.321356 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.321287    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy"
Sep 09 21:56:03.321425 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.321357    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy"
Sep 09 21:56:03.324595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.324578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"
Sep 09 21:56:03.327197 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.327178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 21:56:03.332740 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.332718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"]
Sep 09 21:56:03.490988 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.490950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk\" (UID: \"bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"
Sep 09 21:56:03.592132 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.592032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk\" (UID: \"bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"
Sep 09 21:56:03.592767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.592746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk\" (UID: \"bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"
Sep 09 21:56:03.636072 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.636035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"
Sep 09 21:56:03.907224 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:03.907194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"]
Sep 09 21:56:03.909142 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:03.909113    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc89b1e4_bb61_44cc_89a8_d5cd81a2cfd9.slice/crio-170c02d3bf02750e90ea097524d9c4d2a17fe18f8b38ab6d519ec5a4b688c73a WatchSource:0}: Error finding container 170c02d3bf02750e90ea097524d9c4d2a17fe18f8b38ab6d519ec5a4b688c73a: Status 404 returned error can't find the container with id 170c02d3bf02750e90ea097524d9c4d2a17fe18f8b38ab6d519ec5a4b688c73a
Sep 09 21:56:04.714504 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:04.714064    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9" containerID="bb26d0a7be366ba90648b1b65fdb106e8649592d5e08a773d2031468f7140f40" exitCode=0
Sep 09 21:56:04.714504 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:04.714124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk" event={"ID":"bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9","Type":"ContainerDied","Data":"bb26d0a7be366ba90648b1b65fdb106e8649592d5e08a773d2031468f7140f40"}
Sep 09 21:56:04.714504 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:04.714178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk" event={"ID":"bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9","Type":"ContainerStarted","Data":"170c02d3bf02750e90ea097524d9c4d2a17fe18f8b38ab6d519ec5a4b688c73a"}
Sep 09 21:56:05.862499 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:05.862470    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"
Sep 09 21:56:06.016546 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:06.016466    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9-maas-api-service-ca\") pod \"bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9\" (UID: \"bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9\") "
Sep 09 21:56:06.016831 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:06.016810    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9" (UID: "bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:06.117328 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:06.117293    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:06.727045 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:06.727007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk" event={"ID":"bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9","Type":"ContainerDied","Data":"170c02d3bf02750e90ea097524d9c4d2a17fe18f8b38ab6d519ec5a4b688c73a"}
Sep 09 21:56:06.727219 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:06.727047    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"
Sep 09 21:56:06.727219 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:06.727082    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170c02d3bf02750e90ea097524d9c4d2a17fe18f8b38ab6d519ec5a4b688c73a"
Sep 09 21:56:09.469376 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.469340    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"]
Sep 09 21:56:09.469934 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.469643    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f" podUID="ea64c48f-f42e-4284-95bf-0a3a2cf94f2c" containerName="payload-pre-processing" containerID="cri-o://385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa" gracePeriod=30
Sep 09 21:56:09.700083 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.700016    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"]
Sep 09 21:56:09.702237 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.702209    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e97e70d-tq4qk"]
Sep 09 21:56:09.749242 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.747532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:56:09.749819 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.749792    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea64c48f-f42e-4284-95bf-0a3a2cf94f2c" containerID="385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa" exitCode=0
Sep 09 21:56:09.749971 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.749870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f" event={"ID":"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c","Type":"ContainerDied","Data":"385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa"}
Sep 09 21:56:09.749971 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.749941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f" event={"ID":"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c","Type":"ContainerDied","Data":"6da7e381bba1485078b5e1e89f14344822cc8a96f22234a052cb9a9144321f05"}
Sep 09 21:56:09.750090 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.749979    2579 scope.go:117] "RemoveContainer" containerID="385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa"
Sep 09 21:56:09.761536 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.761513    2579 scope.go:117] "RemoveContainer" containerID="385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa"
Sep 09 21:56:09.761905 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:56:09.761881    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa\": container with ID starting with 385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa not found: ID does not exist" containerID="385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa"
Sep 09 21:56:09.761989 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.761912    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa"} err="failed to get container status \"385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa\": rpc error: code = NotFound desc = could not find container \"385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa\": container with ID starting with 385bf2ed1e491d211390257273930b536c42cd0be10df97a5e943ebb45b122fa not found: ID does not exist"
Sep 09 21:56:09.854914 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.854882    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-plugins-config-volume\") pod \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\" (UID: \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\") "
Sep 09 21:56:09.855123 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.854935    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-kube-api-access-znlgn\") pod \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\" (UID: \"ea64c48f-f42e-4284-95bf-0a3a2cf94f2c\") "
Sep 09 21:56:09.855328 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.855302    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea64c48f-f42e-4284-95bf-0a3a2cf94f2c" (UID: "ea64c48f-f42e-4284-95bf-0a3a2cf94f2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:09.857174 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.857140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-kube-api-access-znlgn" (OuterVolumeSpecName: "kube-api-access-znlgn") pod "ea64c48f-f42e-4284-95bf-0a3a2cf94f2c" (UID: "ea64c48f-f42e-4284-95bf-0a3a2cf94f2c"). InnerVolumeSpecName "kube-api-access-znlgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:09.956520 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.956477    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:09.956520 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:09.956512    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c-kube-api-access-znlgn\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:10.093191 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.093092    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9" path="/var/lib/kubelet/pods/bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9/volumes"
Sep 09 21:56:10.551367 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.551329    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"]
Sep 09 21:56:10.552064 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.552032    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9" containerName="revoke-keys"
Sep 09 21:56:10.552064 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.552058    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9" containerName="revoke-keys"
Sep 09 21:56:10.552214 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.552096    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea64c48f-f42e-4284-95bf-0a3a2cf94f2c" containerName="payload-pre-processing"
Sep 09 21:56:10.552214 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.552106    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea64c48f-f42e-4284-95bf-0a3a2cf94f2c" containerName="payload-pre-processing"
Sep 09 21:56:10.552290 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.552213    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea64c48f-f42e-4284-95bf-0a3a2cf94f2c" containerName="payload-pre-processing"
Sep 09 21:56:10.552290 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.552226    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b12a913-e5ed-4eba-b56f-05f17254e8d1" containerName="istio-proxy"
Sep 09 21:56:10.552290 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.552235    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc89b1e4-bb61-44cc-89a8-d5cd81a2cfd9" containerName="revoke-keys"
Sep 09 21:56:10.556738 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.556713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.559796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.559775    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a691ec3b-openshift-default-dockercfg-c5brm\""
Sep 09 21:56:10.575891 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.572086    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"]
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-envoy\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-data\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjwz\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-kube-api-access-2xjwz\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-podinfo\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-credential-socket\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-token\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-socket\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istiod-ca-cert\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.666622 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.664581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-certs\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.756915 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.756879    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"
Sep 09 21:56:10.765757 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.765724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-envoy\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.765757 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.765759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-data\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766008 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.765780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjwz\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-kube-api-access-2xjwz\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766008 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.765797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-podinfo\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766008 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.765936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-credential-socket\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766008 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-token\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766238 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-socket\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766238 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istiod-ca-cert\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766238 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-certs\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766238 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-data\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766451 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-credential-socket\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766451 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-socket\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766546 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-certs\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.766811 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.766792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istiod-ca-cert\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.768029 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.768002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-envoy\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.768220 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.768202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-podinfo\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.773997 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.773965    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"]
Sep 09 21:56:10.776018 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.775991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-token\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.776403 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.776212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjwz\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-kube-api-access-2xjwz\") pod \"e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") " pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:10.779219 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.779199    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e97e70d-7c5c4d486b-pj95f"]
Sep 09 21:56:10.867140 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:10.867099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:11.147202 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:11.147172    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"]
Sep 09 21:56:11.150093 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:11.150062    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d06fbc2_afc9_4912_bab2_782dcd5672c2.slice/crio-9f5954d9b085e39b423d40e97a9e26fe520f4ada27ec69492df864d894a87157 WatchSource:0}: Error finding container 9f5954d9b085e39b423d40e97a9e26fe520f4ada27ec69492df864d894a87157: Status 404 returned error can't find the container with id 9f5954d9b085e39b423d40e97a9e26fe520f4ada27ec69492df864d894a87157
Sep 09 21:56:11.152450 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:11.152415    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:56:11.152518 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:11.152494    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:56:11.152552 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:11.152523    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:56:11.764150 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:11.764109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" event={"ID":"7d06fbc2-afc9-4912-bab2-782dcd5672c2","Type":"ContainerStarted","Data":"380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e"}
Sep 09 21:56:11.764595 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:11.764170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" event={"ID":"7d06fbc2-afc9-4912-bab2-782dcd5672c2","Type":"ContainerStarted","Data":"9f5954d9b085e39b423d40e97a9e26fe520f4ada27ec69492df864d894a87157"}
Sep 09 21:56:11.789116 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:11.789061    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" podStartSLOduration=1.78904489 podStartE2EDuration="1.78904489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:11.787929859 +0000 UTC m=+1046.210292600" watchObservedRunningTime="2026-09-09 21:56:11.78904489 +0000 UTC m=+1046.211407588"
Sep 09 21:56:11.868001 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:11.867963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:12.093406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:12.093325    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea64c48f-f42e-4284-95bf-0a3a2cf94f2c" path="/var/lib/kubelet/pods/ea64c48f-f42e-4284-95bf-0a3a2cf94f2c/volumes"
Sep 09 21:56:12.873006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:12.872973    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:13.775536 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:13.775500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:13.776554 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:13.776531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:15.449435 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.449392    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"]
Sep 09 21:56:15.462755 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.462719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:15.465505 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.465468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a691ec3b-dockercfg-7zdxq\""
Sep 09 21:56:15.465663 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.465536    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a691ec3b\""
Sep 09 21:56:15.471217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.471187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"]
Sep 09 21:56:15.520519 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.520477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptn9w\" (UniqueName: \"kubernetes.io/projected/89ea2cab-7ef6-4347-ac81-16618adf7d63-kube-api-access-ptn9w\") pod \"payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq\" (UID: \"89ea2cab-7ef6-4347-ac81-16618adf7d63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:15.520952 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.520925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ea2cab-7ef6-4347-ac81-16618adf7d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq\" (UID: \"89ea2cab-7ef6-4347-ac81-16618adf7d63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:15.521072 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.521041    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"]
Sep 09 21:56:15.527140 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.526438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:15.536373 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.536340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"]
Sep 09 21:56:15.622318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.622284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptn9w\" (UniqueName: \"kubernetes.io/projected/89ea2cab-7ef6-4347-ac81-16618adf7d63-kube-api-access-ptn9w\") pod \"payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq\" (UID: \"89ea2cab-7ef6-4347-ac81-16618adf7d63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:15.622505 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.622361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5j7\" (UniqueName: \"kubernetes.io/projected/e9f428b4-3798-4c90-beed-c85a52d1f2d5-kube-api-access-cb5j7\") pod \"payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s\" (UID: \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:15.622505 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.622428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ea2cab-7ef6-4347-ac81-16618adf7d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq\" (UID: \"89ea2cab-7ef6-4347-ac81-16618adf7d63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:15.622505 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.622453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f428b4-3798-4c90-beed-c85a52d1f2d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s\" (UID: \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:15.623168 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.623145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ea2cab-7ef6-4347-ac81-16618adf7d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq\" (UID: \"89ea2cab-7ef6-4347-ac81-16618adf7d63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:15.631004 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.630972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptn9w\" (UniqueName: \"kubernetes.io/projected/89ea2cab-7ef6-4347-ac81-16618adf7d63-kube-api-access-ptn9w\") pod \"payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq\" (UID: \"89ea2cab-7ef6-4347-ac81-16618adf7d63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:15.723119 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.723033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5j7\" (UniqueName: \"kubernetes.io/projected/e9f428b4-3798-4c90-beed-c85a52d1f2d5-kube-api-access-cb5j7\") pod \"payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s\" (UID: \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:15.723119 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.723108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f428b4-3798-4c90-beed-c85a52d1f2d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s\" (UID: \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:15.723817 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.723797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f428b4-3798-4c90-beed-c85a52d1f2d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s\" (UID: \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:15.731581 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.731545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5j7\" (UniqueName: \"kubernetes.io/projected/e9f428b4-3798-4c90-beed-c85a52d1f2d5-kube-api-access-cb5j7\") pod \"payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s\" (UID: \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:15.776374 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.776325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:15.847059 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:15.846960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:16.053930 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.053902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"]
Sep 09 21:56:16.055989 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:16.055958    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ea2cab_7ef6_4347_ac81_16618adf7d63.slice/crio-3b7c917426d6bb37c6c5d9a3227516c62e1c20bb38aa87b16f2b666e4356f63d WatchSource:0}: Error finding container 3b7c917426d6bb37c6c5d9a3227516c62e1c20bb38aa87b16f2b666e4356f63d: Status 404 returned error can't find the container with id 3b7c917426d6bb37c6c5d9a3227516c62e1c20bb38aa87b16f2b666e4356f63d
Sep 09 21:56:16.123225 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.123198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"]
Sep 09 21:56:16.123433 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:16.123411    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f428b4_3798_4c90_beed_c85a52d1f2d5.slice/crio-df021ac97023177df0dfe08eb9abb1021b12322f8201a4566e83fc1b1f679ef4 WatchSource:0}: Error finding container df021ac97023177df0dfe08eb9abb1021b12322f8201a4566e83fc1b1f679ef4: Status 404 returned error can't find the container with id df021ac97023177df0dfe08eb9abb1021b12322f8201a4566e83fc1b1f679ef4
Sep 09 21:56:16.800201 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.800160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s" event={"ID":"e9f428b4-3798-4c90-beed-c85a52d1f2d5","Type":"ContainerStarted","Data":"5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7"}
Sep 09 21:56:16.800671 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.800224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s" event={"ID":"e9f428b4-3798-4c90-beed-c85a52d1f2d5","Type":"ContainerStarted","Data":"df021ac97023177df0dfe08eb9abb1021b12322f8201a4566e83fc1b1f679ef4"}
Sep 09 21:56:16.800671 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.800255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:16.802036 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.802006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq" event={"ID":"89ea2cab-7ef6-4347-ac81-16618adf7d63","Type":"ContainerStarted","Data":"617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99"}
Sep 09 21:56:16.802175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.802057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq" event={"ID":"89ea2cab-7ef6-4347-ac81-16618adf7d63","Type":"ContainerStarted","Data":"3b7c917426d6bb37c6c5d9a3227516c62e1c20bb38aa87b16f2b666e4356f63d"}
Sep 09 21:56:16.802175 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.802099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:16.816490 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.816439    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s" podStartSLOduration=1.8164193929999999 podStartE2EDuration="1.816419393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:16.815166277 +0000 UTC m=+1051.237528978" watchObservedRunningTime="2026-09-09 21:56:16.816419393 +0000 UTC m=+1051.238782098"
Sep 09 21:56:16.831245 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:16.831191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq" podStartSLOduration=1.831172294 podStartE2EDuration="1.831172294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:16.83045673 +0000 UTC m=+1051.252819432" watchObservedRunningTime="2026-09-09 21:56:16.831172294 +0000 UTC m=+1051.253534995"
Sep 09 21:56:27.811900 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:27.811842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:27.813174 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:27.813142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:31.878382 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:31.878341    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"]
Sep 09 21:56:31.882992 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:31.882967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"
Sep 09 21:56:31.886016 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:31.885997    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 21:56:31.891034 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:31.891003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"]
Sep 09 21:56:31.997931 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:31.997876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1098ff3f-8d05-499a-abd9-fd81ae38b8ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-594ab9-t58th\" (UID: \"1098ff3f-8d05-499a-abd9-fd81ae38b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"
Sep 09 21:56:32.098847 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:32.098801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1098ff3f-8d05-499a-abd9-fd81ae38b8ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-594ab9-t58th\" (UID: \"1098ff3f-8d05-499a-abd9-fd81ae38b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"
Sep 09 21:56:32.099430 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:32.099403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1098ff3f-8d05-499a-abd9-fd81ae38b8ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-594ab9-t58th\" (UID: \"1098ff3f-8d05-499a-abd9-fd81ae38b8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"
Sep 09 21:56:32.197793 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:32.197699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"
Sep 09 21:56:32.483662 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:32.483638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"]
Sep 09 21:56:32.486175 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:32.486143    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1098ff3f_8d05_499a_abd9_fd81ae38b8ae.slice/crio-639c1dfed91e6d0100d678a12f9d6fddca9c2435c1cc0f7eb018acf863113021 WatchSource:0}: Error finding container 639c1dfed91e6d0100d678a12f9d6fddca9c2435c1cc0f7eb018acf863113021: Status 404 returned error can't find the container with id 639c1dfed91e6d0100d678a12f9d6fddca9c2435c1cc0f7eb018acf863113021
Sep 09 21:56:32.904165 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:32.904132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th" event={"ID":"1098ff3f-8d05-499a-abd9-fd81ae38b8ae","Type":"ContainerStarted","Data":"639c1dfed91e6d0100d678a12f9d6fddca9c2435c1cc0f7eb018acf863113021"}
Sep 09 21:56:33.912792 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:33.912697    2579 generic.go:356] "Generic (PLEG): container finished" podID="1098ff3f-8d05-499a-abd9-fd81ae38b8ae" containerID="8ea4a0d4183c464c66b2c9bf91d28a77ca4545784840d806b4959c6eaac26da9" exitCode=0
Sep 09 21:56:33.913340 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:33.912797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th" event={"ID":"1098ff3f-8d05-499a-abd9-fd81ae38b8ae","Type":"ContainerDied","Data":"8ea4a0d4183c464c66b2c9bf91d28a77ca4545784840d806b4959c6eaac26da9"}
Sep 09 21:56:33.914831 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:33.914807    2579 generic.go:356] "Generic (PLEG): container finished" podID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerID="380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e" exitCode=0
Sep 09 21:56:33.914967 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:33.914843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" event={"ID":"7d06fbc2-afc9-4912-bab2-782dcd5672c2","Type":"ContainerDied","Data":"380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e"}
Sep 09 21:56:33.915234 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:33.915218    2579 scope.go:117] "RemoveContainer" containerID="380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e"
Sep 09 21:56:33.916445 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:33.916415    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:56:33.916538 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:33.916503    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:56:33.916631 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:33.916552    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:56:34.923195 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:34.923156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" event={"ID":"7d06fbc2-afc9-4912-bab2-782dcd5672c2","Type":"ContainerStarted","Data":"a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809"}
Sep 09 21:56:35.065165 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.065140    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"
Sep 09 21:56:35.135005 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.134963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1098ff3f-8d05-499a-abd9-fd81ae38b8ae-maas-api-service-ca\") pod \"1098ff3f-8d05-499a-abd9-fd81ae38b8ae\" (UID: \"1098ff3f-8d05-499a-abd9-fd81ae38b8ae\") "
Sep 09 21:56:35.136033 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.136007    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1098ff3f-8d05-499a-abd9-fd81ae38b8ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1098ff3f-8d05-499a-abd9-fd81ae38b8ae" (UID: "1098ff3f-8d05-499a-abd9-fd81ae38b8ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:35.236678 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.236553    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1098ff3f-8d05-499a-abd9-fd81ae38b8ae-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:35.867277 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.867240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:35.867491 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.867469    2579 patch_prober.go:28] interesting pod/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 21:56:35.867570 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.867533    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 21:56:35.930886 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.930857    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"
Sep 09 21:56:35.931318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.930886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th" event={"ID":"1098ff3f-8d05-499a-abd9-fd81ae38b8ae","Type":"ContainerDied","Data":"639c1dfed91e6d0100d678a12f9d6fddca9c2435c1cc0f7eb018acf863113021"}
Sep 09 21:56:35.931318 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:35.930936    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="639c1dfed91e6d0100d678a12f9d6fddca9c2435c1cc0f7eb018acf863113021"
Sep 09 21:56:36.868159 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:36.868126    2579 patch_prober.go:28] interesting pod/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 21:56:36.868416 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:36.868194    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 21:56:37.867874 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:37.867837    2579 patch_prober.go:28] interesting pod/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 21:56:37.868279 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:37.867915    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 21:56:38.852112 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.852068    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"]
Sep 09 21:56:38.852576 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.852563    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1098ff3f-8d05-499a-abd9-fd81ae38b8ae" containerName="revoke-keys"
Sep 09 21:56:38.852643 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.852579    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1098ff3f-8d05-499a-abd9-fd81ae38b8ae" containerName="revoke-keys"
Sep 09 21:56:38.852698 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.852687    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1098ff3f-8d05-499a-abd9-fd81ae38b8ae" containerName="revoke-keys"
Sep 09 21:56:38.855052 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.855033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"
Sep 09 21:56:38.857913 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.857887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 21:56:38.863265 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.863240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"]
Sep 09 21:56:38.867442 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.867416    2579 patch_prober.go:28] interesting pod/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 21:56:38.867619 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.867477    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 21:56:38.976743 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:38.976705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e60064f-8be1-4cfc-8ad6-33e3e26d68f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz\" (UID: \"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"
Sep 09 21:56:39.077823 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.077781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e60064f-8be1-4cfc-8ad6-33e3e26d68f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz\" (UID: \"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"
Sep 09 21:56:39.078447 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.078425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e60064f-8be1-4cfc-8ad6-33e3e26d68f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz\" (UID: \"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"
Sep 09 21:56:39.167473 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.167384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"
Sep 09 21:56:39.461994 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.461899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"]
Sep 09 21:56:39.465963 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:39.465933    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e60064f_8be1_4cfc_8ad6_33e3e26d68f2.slice/crio-7dc45d47795556374e911cc78ec8598d4d6abf43e0cffe7d41246abb7e9808f0 WatchSource:0}: Error finding container 7dc45d47795556374e911cc78ec8598d4d6abf43e0cffe7d41246abb7e9808f0: Status 404 returned error can't find the container with id 7dc45d47795556374e911cc78ec8598d4d6abf43e0cffe7d41246abb7e9808f0
Sep 09 21:56:39.867679 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.867639    2579 patch_prober.go:28] interesting pod/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 09 21:56:39.867875 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.867697    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 09 21:56:39.954206 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.954174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz" event={"ID":"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2","Type":"ContainerStarted","Data":"e9a61f2744f9d28fb79f2f784e4f29ef35bd233a7c89227c405e9b6aee7bb225"}
Sep 09 21:56:39.954494 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.954472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz" event={"ID":"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2","Type":"ContainerStarted","Data":"7dc45d47795556374e911cc78ec8598d4d6abf43e0cffe7d41246abb7e9808f0"}
Sep 09 21:56:39.971554 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:39.971465    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz" podStartSLOduration=1.9714520229999999 podStartE2EDuration="1.971452023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:39.969318261 +0000 UTC m=+1074.391680952" watchObservedRunningTime="2026-09-09 21:56:39.971452023 +0000 UTC m=+1074.393814760"
Sep 09 21:56:40.915264 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:40.915217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:40.915766 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:40.915629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:40.916630 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:40.916593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:40.963797 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:40.963749    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e60064f-8be1-4cfc-8ad6-33e3e26d68f2" containerID="e9a61f2744f9d28fb79f2f784e4f29ef35bd233a7c89227c405e9b6aee7bb225" exitCode=0
Sep 09 21:56:40.964198 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:40.964162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz" event={"ID":"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2","Type":"ContainerDied","Data":"e9a61f2744f9d28fb79f2f784e4f29ef35bd233a7c89227c405e9b6aee7bb225"}
Sep 09 21:56:42.114959 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.114935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"
Sep 09 21:56:42.211462 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.211435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e60064f-8be1-4cfc-8ad6-33e3e26d68f2-maas-api-service-ca\") pod \"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2\" (UID: \"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2\") "
Sep 09 21:56:42.211788 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.211766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e60064f-8be1-4cfc-8ad6-33e3e26d68f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e60064f-8be1-4cfc-8ad6-33e3e26d68f2" (UID: "0e60064f-8be1-4cfc-8ad6-33e3e26d68f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:42.313049 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.312963    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e60064f-8be1-4cfc-8ad6-33e3e26d68f2-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:42.649339 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.649305    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"]
Sep 09 21:56:42.649557 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.649533    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8" podUID="d93ac1ff-11ca-4bf2-8027-752aa7ddf77c" containerName="payload-processing" containerID="cri-o://dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c" gracePeriod=30
Sep 09 21:56:42.860365 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.860324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"]
Sep 09 21:56:42.864834 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.864804    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-594ab9-t58th"]
Sep 09 21:56:42.919273 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.919243    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:56:42.978699 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.978665    2579 generic.go:356] "Generic (PLEG): container finished" podID="d93ac1ff-11ca-4bf2-8027-752aa7ddf77c" containerID="dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c" exitCode=0
Sep 09 21:56:42.978884 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.978732    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"
Sep 09 21:56:42.978884 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.978763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8" event={"ID":"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c","Type":"ContainerDied","Data":"dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c"}
Sep 09 21:56:42.978884 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.978840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8" event={"ID":"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c","Type":"ContainerDied","Data":"0d9219fdbc26e49f080fde18c008058f9583c35328213403cd702f76a62db280"}
Sep 09 21:56:42.979058 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.978887    2579 scope.go:117] "RemoveContainer" containerID="dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c"
Sep 09 21:56:42.980474 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.980454    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"
Sep 09 21:56:42.980556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.980453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz" event={"ID":"0e60064f-8be1-4cfc-8ad6-33e3e26d68f2","Type":"ContainerDied","Data":"7dc45d47795556374e911cc78ec8598d4d6abf43e0cffe7d41246abb7e9808f0"}
Sep 09 21:56:42.980617 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.980587    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dc45d47795556374e911cc78ec8598d4d6abf43e0cffe7d41246abb7e9808f0"
Sep 09 21:56:42.990071 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.990051    2579 scope.go:117] "RemoveContainer" containerID="dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c"
Sep 09 21:56:42.990412 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:56:42.990386    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c\": container with ID starting with dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c not found: ID does not exist" containerID="dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c"
Sep 09 21:56:42.990487 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:42.990434    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c"} err="failed to get container status \"dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c\": rpc error: code = NotFound desc = could not find container \"dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c\": container with ID starting with dece00e187f2f9daa97e79cbffed39bbbb38d65d2ecc7b19269cf552aad6075c not found: ID does not exist"
Sep 09 21:56:43.019790 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:43.019734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghp5v\" (UniqueName: \"kubernetes.io/projected/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-kube-api-access-ghp5v\") pod \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\" (UID: \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\") "
Sep 09 21:56:43.020009 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:43.019909    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-plugins-config-volume\") pod \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\" (UID: \"d93ac1ff-11ca-4bf2-8027-752aa7ddf77c\") "
Sep 09 21:56:43.020398 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:43.020370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d93ac1ff-11ca-4bf2-8027-752aa7ddf77c" (UID: "d93ac1ff-11ca-4bf2-8027-752aa7ddf77c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:43.022718 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:43.022634    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-kube-api-access-ghp5v" (OuterVolumeSpecName: "kube-api-access-ghp5v") pod "d93ac1ff-11ca-4bf2-8027-752aa7ddf77c" (UID: "d93ac1ff-11ca-4bf2-8027-752aa7ddf77c"). InnerVolumeSpecName "kube-api-access-ghp5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:43.121531 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:43.121487    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.121531 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:43.121522    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghp5v\" (UniqueName: \"kubernetes.io/projected/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c-kube-api-access-ghp5v\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:43.304674 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:43.304630    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"]
Sep 09 21:56:43.308895 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:43.308866    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-594ab9-67f8544b65-bf2c8"]
Sep 09 21:56:44.093548 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:44.093513    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1098ff3f-8d05-499a-abd9-fd81ae38b8ae" path="/var/lib/kubelet/pods/1098ff3f-8d05-499a-abd9-fd81ae38b8ae/volumes"
Sep 09 21:56:44.093996 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:44.093971    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d93ac1ff-11ca-4bf2-8027-752aa7ddf77c" path="/var/lib/kubelet/pods/d93ac1ff-11ca-4bf2-8027-752aa7ddf77c/volumes"
Sep 09 21:56:45.525010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.524971    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"]
Sep 09 21:56:45.525481 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.525253    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s" podUID="e9f428b4-3798-4c90-beed-c85a52d1f2d5" containerName="payload-processing" containerID="cri-o://5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7" gracePeriod=30
Sep 09 21:56:45.543808 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.543778    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"]
Sep 09 21:56:45.544002 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.543982    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq" podUID="89ea2cab-7ef6-4347-ac81-16618adf7d63" containerName="payload-pre-processing" containerID="cri-o://617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99" gracePeriod=30
Sep 09 21:56:45.844585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.844551    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:45.845749 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.845637    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:45.904225 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.904152    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"]
Sep 09 21:56:45.908748 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.908719    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a691ec3b-kblcz"]
Sep 09 21:56:45.952300 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.952270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ea2cab-7ef6-4347-ac81-16618adf7d63-plugins-config-volume\") pod \"89ea2cab-7ef6-4347-ac81-16618adf7d63\" (UID: \"89ea2cab-7ef6-4347-ac81-16618adf7d63\") "
Sep 09 21:56:45.952300 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.952301    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f428b4-3798-4c90-beed-c85a52d1f2d5-plugins-config-volume\") pod \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\" (UID: \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\") "
Sep 09 21:56:45.952513 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.952336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb5j7\" (UniqueName: \"kubernetes.io/projected/e9f428b4-3798-4c90-beed-c85a52d1f2d5-kube-api-access-cb5j7\") pod \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\" (UID: \"e9f428b4-3798-4c90-beed-c85a52d1f2d5\") "
Sep 09 21:56:45.952513 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.952388    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptn9w\" (UniqueName: \"kubernetes.io/projected/89ea2cab-7ef6-4347-ac81-16618adf7d63-kube-api-access-ptn9w\") pod \"89ea2cab-7ef6-4347-ac81-16618adf7d63\" (UID: \"89ea2cab-7ef6-4347-ac81-16618adf7d63\") "
Sep 09 21:56:45.952744 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.952718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f428b4-3798-4c90-beed-c85a52d1f2d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9f428b4-3798-4c90-beed-c85a52d1f2d5" (UID: "e9f428b4-3798-4c90-beed-c85a52d1f2d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:45.952828 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.952739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ea2cab-7ef6-4347-ac81-16618adf7d63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89ea2cab-7ef6-4347-ac81-16618adf7d63" (UID: "89ea2cab-7ef6-4347-ac81-16618adf7d63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:45.954389 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.954363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f428b4-3798-4c90-beed-c85a52d1f2d5-kube-api-access-cb5j7" (OuterVolumeSpecName: "kube-api-access-cb5j7") pod "e9f428b4-3798-4c90-beed-c85a52d1f2d5" (UID: "e9f428b4-3798-4c90-beed-c85a52d1f2d5"). InnerVolumeSpecName "kube-api-access-cb5j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:45.954462 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:45.954397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ea2cab-7ef6-4347-ac81-16618adf7d63-kube-api-access-ptn9w" (OuterVolumeSpecName: "kube-api-access-ptn9w") pod "89ea2cab-7ef6-4347-ac81-16618adf7d63" (UID: "89ea2cab-7ef6-4347-ac81-16618adf7d63"). InnerVolumeSpecName "kube-api-access-ptn9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:46.000486 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.000456    2579 generic.go:356] "Generic (PLEG): container finished" podID="89ea2cab-7ef6-4347-ac81-16618adf7d63" containerID="617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99" exitCode=0
Sep 09 21:56:46.000698 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.000507    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"
Sep 09 21:56:46.000698 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.000529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq" event={"ID":"89ea2cab-7ef6-4347-ac81-16618adf7d63","Type":"ContainerDied","Data":"617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99"}
Sep 09 21:56:46.000698 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.000580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq" event={"ID":"89ea2cab-7ef6-4347-ac81-16618adf7d63","Type":"ContainerDied","Data":"3b7c917426d6bb37c6c5d9a3227516c62e1c20bb38aa87b16f2b666e4356f63d"}
Sep 09 21:56:46.000698 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.000616    2579 scope.go:117] "RemoveContainer" containerID="617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99"
Sep 09 21:56:46.002348 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.002278    2579 generic.go:356] "Generic (PLEG): container finished" podID="e9f428b4-3798-4c90-beed-c85a52d1f2d5" containerID="5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7" exitCode=0
Sep 09 21:56:46.002348 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.002316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"
Sep 09 21:56:46.002509 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.002312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s" event={"ID":"e9f428b4-3798-4c90-beed-c85a52d1f2d5","Type":"ContainerDied","Data":"5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7"}
Sep 09 21:56:46.002509 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.002433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s" event={"ID":"e9f428b4-3798-4c90-beed-c85a52d1f2d5","Type":"ContainerDied","Data":"df021ac97023177df0dfe08eb9abb1021b12322f8201a4566e83fc1b1f679ef4"}
Sep 09 21:56:46.013036 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.012955    2579 scope.go:117] "RemoveContainer" containerID="617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99"
Sep 09 21:56:46.013338 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:56:46.013304    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99\": container with ID starting with 617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99 not found: ID does not exist" containerID="617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99"
Sep 09 21:56:46.013444 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.013345    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99"} err="failed to get container status \"617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99\": rpc error: code = NotFound desc = could not find container \"617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99\": container with ID starting with 617fd61805aebf23e2f67efdc66268211053b4d68f6f3f45fd2658b068545f99 not found: ID does not exist"
Sep 09 21:56:46.013444 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.013368    2579 scope.go:117] "RemoveContainer" containerID="5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7"
Sep 09 21:56:46.022040 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.022020    2579 scope.go:117] "RemoveContainer" containerID="5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7"
Sep 09 21:56:46.022337 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:56:46.022319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7\": container with ID starting with 5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7 not found: ID does not exist" containerID="5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7"
Sep 09 21:56:46.022393 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.022345    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7"} err="failed to get container status \"5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7\": rpc error: code = NotFound desc = could not find container \"5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7\": container with ID starting with 5f8974e660b22d1a720c1b3f63297998743c82ad73221378bcd706688af322c7 not found: ID does not exist"
Sep 09 21:56:46.044812 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.044780    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"]
Sep 09 21:56:46.046735 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.046714    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a691ec3b-5c9969774-gwdkq"]
Sep 09 21:56:46.054173 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.054126    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5j7\" (UniqueName: \"kubernetes.io/projected/e9f428b4-3798-4c90-beed-c85a52d1f2d5-kube-api-access-cb5j7\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:46.054173 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.054147    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptn9w\" (UniqueName: \"kubernetes.io/projected/89ea2cab-7ef6-4347-ac81-16618adf7d63-kube-api-access-ptn9w\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:46.054173 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.054157    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89ea2cab-7ef6-4347-ac81-16618adf7d63-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:46.054173 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.054165    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f428b4-3798-4c90-beed-c85a52d1f2d5-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:46.061987 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.061953    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"]
Sep 09 21:56:46.063959 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.063940    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a691ec3b-8656459879-fxs5s"]
Sep 09 21:56:46.092467 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.092442    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e60064f-8be1-4cfc-8ad6-33e3e26d68f2" path="/var/lib/kubelet/pods/0e60064f-8be1-4cfc-8ad6-33e3e26d68f2/volumes"
Sep 09 21:56:46.092850 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.092836    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89ea2cab-7ef6-4347-ac81-16618adf7d63" path="/var/lib/kubelet/pods/89ea2cab-7ef6-4347-ac81-16618adf7d63/volumes"
Sep 09 21:56:46.093153 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:46.093141    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f428b4-3798-4c90-beed-c85a52d1f2d5" path="/var/lib/kubelet/pods/e9f428b4-3798-4c90-beed-c85a52d1f2d5/volumes"
Sep 09 21:56:47.052871 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:47.052836    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"]
Sep 09 21:56:47.053357 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:47.053048    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy" containerID="cri-o://a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809" gracePeriod=30
Sep 09 21:56:52.322443 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.322373    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:52.418363 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418330    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-credential-socket\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.418526 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418388    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xjwz\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-kube-api-access-2xjwz\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.418526 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-podinfo\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.418526 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-socket\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.418730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-envoy\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.418730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418566    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-data\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.418730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418615    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-token\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.418730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418646    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-certs\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.418730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418677    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:52.418730 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istiod-ca-cert\") pod \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\" (UID: \"7d06fbc2-afc9-4912-bab2-782dcd5672c2\") "
Sep 09 21:56:52.419028 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418946    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:52.419028 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.418957    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:52.419127 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.419102    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:52.419127 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.419120    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:52.419228 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.419134    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:52.419228 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.419159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:52.419333 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.419252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:52.420847 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.420819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-kube-api-access-2xjwz" (OuterVolumeSpecName: "kube-api-access-2xjwz") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "kube-api-access-2xjwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:52.420982 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.420871    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:56:52.421185 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.421166    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:52.421238 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.421176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d06fbc2-afc9-4912-bab2-782dcd5672c2" (UID: "7d06fbc2-afc9-4912-bab2-782dcd5672c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:52.520473 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.520435    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xjwz\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-kube-api-access-2xjwz\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:52.520735 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.520716    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:52.520836 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.520824    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:52.520923 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.520913    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:52.520997 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.520987    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d06fbc2-afc9-4912-bab2-782dcd5672c2-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:52.521063 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:52.521055    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d06fbc2-afc9-4912-bab2-782dcd5672c2-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:56:53.005479 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.005445    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"]
Sep 09 21:56:53.005943 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.005928    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f428b4-3798-4c90-beed-c85a52d1f2d5" containerName="payload-processing"
Sep 09 21:56:53.006006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.005944    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f428b4-3798-4c90-beed-c85a52d1f2d5" containerName="payload-processing"
Sep 09 21:56:53.006006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.005954    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93ac1ff-11ca-4bf2-8027-752aa7ddf77c" containerName="payload-processing"
Sep 09 21:56:53.006006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.005959    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93ac1ff-11ca-4bf2-8027-752aa7ddf77c" containerName="payload-processing"
Sep 09 21:56:53.006006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.005967    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e60064f-8be1-4cfc-8ad6-33e3e26d68f2" containerName="revoke-keys"
Sep 09 21:56:53.006006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.005973    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e60064f-8be1-4cfc-8ad6-33e3e26d68f2" containerName="revoke-keys"
Sep 09 21:56:53.006006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006000    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy"
Sep 09 21:56:53.006006 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006006    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006014    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ea2cab-7ef6-4347-ac81-16618adf7d63" containerName="payload-pre-processing"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006019    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ea2cab-7ef6-4347-ac81-16618adf7d63" containerName="payload-pre-processing"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006027    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006032    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006098    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006105    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ea2cab-7ef6-4347-ac81-16618adf7d63" containerName="payload-pre-processing"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006115    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f428b4-3798-4c90-beed-c85a52d1f2d5" containerName="payload-processing"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006120    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerName="istio-proxy"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006128    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e60064f-8be1-4cfc-8ad6-33e3e26d68f2" containerName="revoke-keys"
Sep 09 21:56:53.006217 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.006134    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93ac1ff-11ca-4bf2-8027-752aa7ddf77c" containerName="payload-processing"
Sep 09 21:56:53.008361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.008345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:53.011184 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.011160    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5677f8b9-dockercfg-wklhw\""
Sep 09 21:56:53.011399 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.011384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5677f8b9\""
Sep 09 21:56:53.022521 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.022496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"]
Sep 09 21:56:53.024747 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.024723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbbp\" (UniqueName: \"kubernetes.io/projected/41bae968-d1ce-4e82-af38-1edd92d906aa-kube-api-access-nzbbp\") pod \"payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx\" (UID: \"41bae968-d1ce-4e82-af38-1edd92d906aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:53.024845 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.024826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bae968-d1ce-4e82-af38-1edd92d906aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx\" (UID: \"41bae968-d1ce-4e82-af38-1edd92d906aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:53.056215 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.056182    2579 generic.go:356] "Generic (PLEG): container finished" podID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" containerID="a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809" exitCode=0
Sep 09 21:56:53.056388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.056218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" event={"ID":"7d06fbc2-afc9-4912-bab2-782dcd5672c2","Type":"ContainerDied","Data":"a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809"}
Sep 09 21:56:53.056388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.056258    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"
Sep 09 21:56:53.056388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.056275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf" event={"ID":"7d06fbc2-afc9-4912-bab2-782dcd5672c2","Type":"ContainerDied","Data":"9f5954d9b085e39b423d40e97a9e26fe520f4ada27ec69492df864d894a87157"}
Sep 09 21:56:53.056388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.056302    2579 scope.go:117] "RemoveContainer" containerID="a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809"
Sep 09 21:56:53.066241 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.066223    2579 scope.go:117] "RemoveContainer" containerID="380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e"
Sep 09 21:56:53.074621 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.074582    2579 scope.go:117] "RemoveContainer" containerID="a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809"
Sep 09 21:56:53.074916 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:56:53.074897    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809\": container with ID starting with a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809 not found: ID does not exist" containerID="a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809"
Sep 09 21:56:53.074986 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.074925    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809"} err="failed to get container status \"a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809\": rpc error: code = NotFound desc = could not find container \"a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809\": container with ID starting with a82480e60defb3c6a2b953d0d80eedc29544df5451e09e84110160d91736b809 not found: ID does not exist"
Sep 09 21:56:53.074986 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.074942    2579 scope.go:117] "RemoveContainer" containerID="380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e"
Sep 09 21:56:53.075204 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:56:53.075187    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e\": container with ID starting with 380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e not found: ID does not exist" containerID="380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e"
Sep 09 21:56:53.075242 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.075208    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e"} err="failed to get container status \"380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e\": rpc error: code = NotFound desc = could not find container \"380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e\": container with ID starting with 380d6a7fe13cdcdcfc55deca0ff37806f1b2558caf1580399a7bc60cad3fa30e not found: ID does not exist"
Sep 09 21:56:53.090470 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.090446    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"]
Sep 09 21:56:53.093893 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.093875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:53.099599 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.099572    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"]
Sep 09 21:56:53.107275 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.107251    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a691ec3b-openshift-default-596b65485b-8s7jf"]
Sep 09 21:56:53.126288 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.126265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbbp\" (UniqueName: \"kubernetes.io/projected/41bae968-d1ce-4e82-af38-1edd92d906aa-kube-api-access-nzbbp\") pod \"payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx\" (UID: \"41bae968-d1ce-4e82-af38-1edd92d906aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:53.126418 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.126371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bae968-d1ce-4e82-af38-1edd92d906aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx\" (UID: \"41bae968-d1ce-4e82-af38-1edd92d906aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:53.126469 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.126439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae71806-0a4e-46d3-8dec-89eb72721b69-plugins-config-volume\") pod \"payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv\" (UID: \"aae71806-0a4e-46d3-8dec-89eb72721b69\") " pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:53.126520 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.126471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvkc\" (UniqueName: \"kubernetes.io/projected/aae71806-0a4e-46d3-8dec-89eb72721b69-kube-api-access-8xvkc\") pod \"payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv\" (UID: \"aae71806-0a4e-46d3-8dec-89eb72721b69\") " pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:53.127252 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.127229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bae968-d1ce-4e82-af38-1edd92d906aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx\" (UID: \"41bae968-d1ce-4e82-af38-1edd92d906aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:53.131718 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.131691    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"]
Sep 09 21:56:53.145079 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.145053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbbp\" (UniqueName: \"kubernetes.io/projected/41bae968-d1ce-4e82-af38-1edd92d906aa-kube-api-access-nzbbp\") pod \"payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx\" (UID: \"41bae968-d1ce-4e82-af38-1edd92d906aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:53.227957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.227922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae71806-0a4e-46d3-8dec-89eb72721b69-plugins-config-volume\") pod \"payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv\" (UID: \"aae71806-0a4e-46d3-8dec-89eb72721b69\") " pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:53.227957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.227962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvkc\" (UniqueName: \"kubernetes.io/projected/aae71806-0a4e-46d3-8dec-89eb72721b69-kube-api-access-8xvkc\") pod \"payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv\" (UID: \"aae71806-0a4e-46d3-8dec-89eb72721b69\") " pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:53.228584 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.228563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae71806-0a4e-46d3-8dec-89eb72721b69-plugins-config-volume\") pod \"payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv\" (UID: \"aae71806-0a4e-46d3-8dec-89eb72721b69\") " pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:53.239744 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.239712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvkc\" (UniqueName: \"kubernetes.io/projected/aae71806-0a4e-46d3-8dec-89eb72721b69-kube-api-access-8xvkc\") pod \"payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv\" (UID: \"aae71806-0a4e-46d3-8dec-89eb72721b69\") " pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:53.318933 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.318840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:53.406135 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.406103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:53.626301 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.626277    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"]
Sep 09 21:56:53.628462 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:53.628438    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bae968_d1ce_4e82_af38_1edd92d906aa.slice/crio-453bb3ae8ecaeed578032322a34fc89564640eff7375252616f03d13875a3129 WatchSource:0}: Error finding container 453bb3ae8ecaeed578032322a34fc89564640eff7375252616f03d13875a3129: Status 404 returned error can't find the container with id 453bb3ae8ecaeed578032322a34fc89564640eff7375252616f03d13875a3129
Sep 09 21:56:53.688299 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:53.688273    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"]
Sep 09 21:56:53.691051 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:53.691026    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae71806_0a4e_46d3_8dec_89eb72721b69.slice/crio-3bde5e9bc2893f57d759fb3c51c9710fd94151ea7ec1c62e82e203b8c96710d8 WatchSource:0}: Error finding container 3bde5e9bc2893f57d759fb3c51c9710fd94151ea7ec1c62e82e203b8c96710d8: Status 404 returned error can't find the container with id 3bde5e9bc2893f57d759fb3c51c9710fd94151ea7ec1c62e82e203b8c96710d8
Sep 09 21:56:54.064388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.064289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv" event={"ID":"aae71806-0a4e-46d3-8dec-89eb72721b69","Type":"ContainerStarted","Data":"29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92"}
Sep 09 21:56:54.064388 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.064349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv" event={"ID":"aae71806-0a4e-46d3-8dec-89eb72721b69","Type":"ContainerStarted","Data":"3bde5e9bc2893f57d759fb3c51c9710fd94151ea7ec1c62e82e203b8c96710d8"}
Sep 09 21:56:54.064592 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.064413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:56:54.065999 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.065960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx" event={"ID":"41bae968-d1ce-4e82-af38-1edd92d906aa","Type":"ContainerStarted","Data":"64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03"}
Sep 09 21:56:54.066136 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.066021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx" event={"ID":"41bae968-d1ce-4e82-af38-1edd92d906aa","Type":"ContainerStarted","Data":"453bb3ae8ecaeed578032322a34fc89564640eff7375252616f03d13875a3129"}
Sep 09 21:56:54.066136 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.066052    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:56:54.093834 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.093793    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d06fbc2-afc9-4912-bab2-782dcd5672c2" path="/var/lib/kubelet/pods/7d06fbc2-afc9-4912-bab2-782dcd5672c2/volumes"
Sep 09 21:56:54.095064 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.095014    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv" podStartSLOduration=1.09499901 podStartE2EDuration="1.09499901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:54.092590168 +0000 UTC m=+1088.514952870" watchObservedRunningTime="2026-09-09 21:56:54.09499901 +0000 UTC m=+1088.517361761"
Sep 09 21:56:54.114508 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:54.114457    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx" podStartSLOduration=2.114440501 podStartE2EDuration="2.114440501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:54.111246982 +0000 UTC m=+1088.533609682" watchObservedRunningTime="2026-09-09 21:56:54.114440501 +0000 UTC m=+1088.536803201"
Sep 09 21:56:56.463491 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.463453    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"]
Sep 09 21:56:56.468390 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.468360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:56.471633 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.471586    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-bbdd85\""
Sep 09 21:56:56.486233 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.486060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"]
Sep 09 21:56:56.496424 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.496373    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"]
Sep 09 21:56:56.499841 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.499812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:56.527680 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.527639    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"]
Sep 09 21:56:56.564809 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.564772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gqj\" (UniqueName: \"kubernetes.io/projected/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-kube-api-access-f8gqj\") pod \"payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z\" (UID: \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:56.565010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.564826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z\" (UID: \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:56.565010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.564892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37012b2-4d6f-49ab-948e-26fe35dea7a1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r\" (UID: \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:56.565010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.564928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8prf\" (UniqueName: \"kubernetes.io/projected/a37012b2-4d6f-49ab-948e-26fe35dea7a1-kube-api-access-b8prf\") pod \"payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r\" (UID: \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:56.665678 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.665641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gqj\" (UniqueName: \"kubernetes.io/projected/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-kube-api-access-f8gqj\") pod \"payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z\" (UID: \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:56.665896 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.665687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z\" (UID: \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:56.665896 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.665735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37012b2-4d6f-49ab-948e-26fe35dea7a1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r\" (UID: \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:56.665896 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.665762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8prf\" (UniqueName: \"kubernetes.io/projected/a37012b2-4d6f-49ab-948e-26fe35dea7a1-kube-api-access-b8prf\") pod \"payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r\" (UID: \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:56.666511 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.666487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z\" (UID: \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:56.666570 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.666488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37012b2-4d6f-49ab-948e-26fe35dea7a1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r\" (UID: \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:56.682265 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.682227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8prf\" (UniqueName: \"kubernetes.io/projected/a37012b2-4d6f-49ab-948e-26fe35dea7a1-kube-api-access-b8prf\") pod \"payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r\" (UID: \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:56.682265 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.682260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gqj\" (UniqueName: \"kubernetes.io/projected/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-kube-api-access-f8gqj\") pod \"payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z\" (UID: \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:56.782061 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.781940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:56.814192 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:56.814159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:57.057094 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:57.057006    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"]
Sep 09 21:56:57.059990 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:57.059961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bd80d3_3476_48a0_9140_29ecd5d7ffe0.slice/crio-73ab8a5a4af7d48d24e9ac4ac65cb76ac75c034fcc75dd7ebf63206c7311e09a WatchSource:0}: Error finding container 73ab8a5a4af7d48d24e9ac4ac65cb76ac75c034fcc75dd7ebf63206c7311e09a: Status 404 returned error can't find the container with id 73ab8a5a4af7d48d24e9ac4ac65cb76ac75c034fcc75dd7ebf63206c7311e09a
Sep 09 21:56:57.086016 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:57.085988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z" event={"ID":"63bd80d3-3476-48a0-9140-29ecd5d7ffe0","Type":"ContainerStarted","Data":"73ab8a5a4af7d48d24e9ac4ac65cb76ac75c034fcc75dd7ebf63206c7311e09a"}
Sep 09 21:56:57.110574 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:57.110278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"]
Sep 09 21:56:57.114675 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:56:57.114647    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37012b2_4d6f_49ab_948e_26fe35dea7a1.slice/crio-e8072929408ee3798ad325737e5aa1de643723d1299080b82ad05feccbb4290b WatchSource:0}: Error finding container e8072929408ee3798ad325737e5aa1de643723d1299080b82ad05feccbb4290b: Status 404 returned error can't find the container with id e8072929408ee3798ad325737e5aa1de643723d1299080b82ad05feccbb4290b
Sep 09 21:56:58.095463 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:58.095426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r" event={"ID":"a37012b2-4d6f-49ab-948e-26fe35dea7a1","Type":"ContainerStarted","Data":"08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72"}
Sep 09 21:56:58.095955 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:58.095486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r" event={"ID":"a37012b2-4d6f-49ab-948e-26fe35dea7a1","Type":"ContainerStarted","Data":"e8072929408ee3798ad325737e5aa1de643723d1299080b82ad05feccbb4290b"}
Sep 09 21:56:58.095955 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:58.095590    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:56:58.097082 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:58.097056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z" event={"ID":"63bd80d3-3476-48a0-9140-29ecd5d7ffe0","Type":"ContainerStarted","Data":"c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d"}
Sep 09 21:56:58.097167 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:58.097115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:56:58.125895 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:58.125838    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r" podStartSLOduration=2.125820149 podStartE2EDuration="2.125820149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:58.125046244 +0000 UTC m=+1092.547408937" watchObservedRunningTime="2026-09-09 21:56:58.125820149 +0000 UTC m=+1092.548182849"
Sep 09 21:56:58.161252 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:56:58.161196    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z" podStartSLOduration=2.161179744 podStartE2EDuration="2.161179744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:58.159122637 +0000 UTC m=+1092.581485337" watchObservedRunningTime="2026-09-09 21:56:58.161179744 +0000 UTC m=+1092.583542444"
Sep 09 21:57:05.074559 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:05.074526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:57:05.074968 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:05.074630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:57:09.106532 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:09.106492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:57:09.107122 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:09.106805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:57:20.601459 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:20.601421    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"]
Sep 09 21:57:20.605171 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:20.605147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"
Sep 09 21:57:20.607751 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:20.607729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 21:57:20.616120 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:20.616090    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"]
Sep 09 21:57:20.708669 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:20.708636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc3f484-3da7-41af-8a9c-c78fb664041f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8\" (UID: \"acc3f484-3da7-41af-8a9c-c78fb664041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"
Sep 09 21:57:20.809576 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:20.809543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc3f484-3da7-41af-8a9c-c78fb664041f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8\" (UID: \"acc3f484-3da7-41af-8a9c-c78fb664041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"
Sep 09 21:57:20.810300 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:20.810276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc3f484-3da7-41af-8a9c-c78fb664041f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8\" (UID: \"acc3f484-3da7-41af-8a9c-c78fb664041f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"
Sep 09 21:57:20.917481 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:20.917383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"
Sep 09 21:57:21.187015 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:21.186990    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"]
Sep 09 21:57:21.188915 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:57:21.188888    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc3f484_3da7_41af_8a9c_c78fb664041f.slice/crio-bd06c23df3b75465fc21bc181f69fe38548a4215527f1ee8dc42e516d4df27c9 WatchSource:0}: Error finding container bd06c23df3b75465fc21bc181f69fe38548a4215527f1ee8dc42e516d4df27c9: Status 404 returned error can't find the container with id bd06c23df3b75465fc21bc181f69fe38548a4215527f1ee8dc42e516d4df27c9
Sep 09 21:57:21.229045 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:21.229015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8" event={"ID":"acc3f484-3da7-41af-8a9c-c78fb664041f","Type":"ContainerStarted","Data":"bd06c23df3b75465fc21bc181f69fe38548a4215527f1ee8dc42e516d4df27c9"}
Sep 09 21:57:22.236691 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:22.236585    2579 generic.go:356] "Generic (PLEG): container finished" podID="acc3f484-3da7-41af-8a9c-c78fb664041f" containerID="ca5d1d17783b2fa5c47cf1aaa4a67268bc0010bc5e4217fd5d562866f6a7294e" exitCode=0
Sep 09 21:57:22.236691 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:22.236634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8" event={"ID":"acc3f484-3da7-41af-8a9c-c78fb664041f","Type":"ContainerDied","Data":"ca5d1d17783b2fa5c47cf1aaa4a67268bc0010bc5e4217fd5d562866f6a7294e"}
Sep 09 21:57:23.408771 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:23.408740    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"
Sep 09 21:57:23.537501 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:23.537399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc3f484-3da7-41af-8a9c-c78fb664041f-maas-api-service-ca\") pod \"acc3f484-3da7-41af-8a9c-c78fb664041f\" (UID: \"acc3f484-3da7-41af-8a9c-c78fb664041f\") "
Sep 09 21:57:23.537750 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:23.537727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc3f484-3da7-41af-8a9c-c78fb664041f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "acc3f484-3da7-41af-8a9c-c78fb664041f" (UID: "acc3f484-3da7-41af-8a9c-c78fb664041f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:57:23.537874 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:23.537859    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc3f484-3da7-41af-8a9c-c78fb664041f-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:24.251070 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:24.251040    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"
Sep 09 21:57:24.251244 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:24.251081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8" event={"ID":"acc3f484-3da7-41af-8a9c-c78fb664041f","Type":"ContainerDied","Data":"bd06c23df3b75465fc21bc181f69fe38548a4215527f1ee8dc42e516d4df27c9"}
Sep 09 21:57:24.251244 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:24.251135    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd06c23df3b75465fc21bc181f69fe38548a4215527f1ee8dc42e516d4df27c9"
Sep 09 21:57:32.791924 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:32.791889    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"]
Sep 09 21:57:32.792414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:32.792103    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv" podUID="aae71806-0a4e-46d3-8dec-89eb72721b69" containerName="payload-processing" containerID="cri-o://29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92" gracePeriod=30
Sep 09 21:57:32.800215 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:32.800173    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"]
Sep 09 21:57:32.800866 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:32.800420    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx" podUID="41bae968-d1ce-4e82-af38-1edd92d906aa" containerName="payload-pre-processing" containerID="cri-o://64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03" gracePeriod=30
Sep 09 21:57:32.972242 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:32.972161    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"]
Sep 09 21:57:32.976574 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:32.976544    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5677f8b9-9hmn8"]
Sep 09 21:57:33.098917 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.098454    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:57:33.100587 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.100561    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:57:33.135559 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.135529    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xvkc\" (UniqueName: \"kubernetes.io/projected/aae71806-0a4e-46d3-8dec-89eb72721b69-kube-api-access-8xvkc\") pod \"aae71806-0a4e-46d3-8dec-89eb72721b69\" (UID: \"aae71806-0a4e-46d3-8dec-89eb72721b69\") "
Sep 09 21:57:33.135778 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.135655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzbbp\" (UniqueName: \"kubernetes.io/projected/41bae968-d1ce-4e82-af38-1edd92d906aa-kube-api-access-nzbbp\") pod \"41bae968-d1ce-4e82-af38-1edd92d906aa\" (UID: \"41bae968-d1ce-4e82-af38-1edd92d906aa\") "
Sep 09 21:57:33.135778 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.135675    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bae968-d1ce-4e82-af38-1edd92d906aa-plugins-config-volume\") pod \"41bae968-d1ce-4e82-af38-1edd92d906aa\" (UID: \"41bae968-d1ce-4e82-af38-1edd92d906aa\") "
Sep 09 21:57:33.135778 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.135769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae71806-0a4e-46d3-8dec-89eb72721b69-plugins-config-volume\") pod \"aae71806-0a4e-46d3-8dec-89eb72721b69\" (UID: \"aae71806-0a4e-46d3-8dec-89eb72721b69\") "
Sep 09 21:57:33.136189 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.136161    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae71806-0a4e-46d3-8dec-89eb72721b69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aae71806-0a4e-46d3-8dec-89eb72721b69" (UID: "aae71806-0a4e-46d3-8dec-89eb72721b69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:57:33.136827 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.136161    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bae968-d1ce-4e82-af38-1edd92d906aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41bae968-d1ce-4e82-af38-1edd92d906aa" (UID: "41bae968-d1ce-4e82-af38-1edd92d906aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:57:33.138294 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.138227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae71806-0a4e-46d3-8dec-89eb72721b69-kube-api-access-8xvkc" (OuterVolumeSpecName: "kube-api-access-8xvkc") pod "aae71806-0a4e-46d3-8dec-89eb72721b69" (UID: "aae71806-0a4e-46d3-8dec-89eb72721b69"). InnerVolumeSpecName "kube-api-access-8xvkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:57:33.139194 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.139165    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bae968-d1ce-4e82-af38-1edd92d906aa-kube-api-access-nzbbp" (OuterVolumeSpecName: "kube-api-access-nzbbp") pod "41bae968-d1ce-4e82-af38-1edd92d906aa" (UID: "41bae968-d1ce-4e82-af38-1edd92d906aa"). InnerVolumeSpecName "kube-api-access-nzbbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:57:33.237092 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.237050    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae71806-0a4e-46d3-8dec-89eb72721b69-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:33.237092 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.237082    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvkc\" (UniqueName: \"kubernetes.io/projected/aae71806-0a4e-46d3-8dec-89eb72721b69-kube-api-access-8xvkc\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:33.237092 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.237095    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbbp\" (UniqueName: \"kubernetes.io/projected/41bae968-d1ce-4e82-af38-1edd92d906aa-kube-api-access-nzbbp\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:33.237092 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.237103    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bae968-d1ce-4e82-af38-1edd92d906aa-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:33.305859 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.305753    2579 generic.go:356] "Generic (PLEG): container finished" podID="aae71806-0a4e-46d3-8dec-89eb72721b69" containerID="29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92" exitCode=0
Sep 09 21:57:33.305859 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.305800    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"
Sep 09 21:57:33.306107 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.305875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv" event={"ID":"aae71806-0a4e-46d3-8dec-89eb72721b69","Type":"ContainerDied","Data":"29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92"}
Sep 09 21:57:33.306107 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.305950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv" event={"ID":"aae71806-0a4e-46d3-8dec-89eb72721b69","Type":"ContainerDied","Data":"3bde5e9bc2893f57d759fb3c51c9710fd94151ea7ec1c62e82e203b8c96710d8"}
Sep 09 21:57:33.306107 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.305993    2579 scope.go:117] "RemoveContainer" containerID="29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92"
Sep 09 21:57:33.307446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.307421    2579 generic.go:356] "Generic (PLEG): container finished" podID="41bae968-d1ce-4e82-af38-1edd92d906aa" containerID="64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03" exitCode=0
Sep 09 21:57:33.307575 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.307557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx" event={"ID":"41bae968-d1ce-4e82-af38-1edd92d906aa","Type":"ContainerDied","Data":"64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03"}
Sep 09 21:57:33.307654 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.307636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx" event={"ID":"41bae968-d1ce-4e82-af38-1edd92d906aa","Type":"ContainerDied","Data":"453bb3ae8ecaeed578032322a34fc89564640eff7375252616f03d13875a3129"}
Sep 09 21:57:33.308060 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.308027    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"
Sep 09 21:57:33.317085 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.317060    2579 scope.go:117] "RemoveContainer" containerID="29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92"
Sep 09 21:57:33.317434 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:57:33.317396    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92\": container with ID starting with 29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92 not found: ID does not exist" containerID="29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92"
Sep 09 21:57:33.317512 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.317427    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92"} err="failed to get container status \"29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92\": rpc error: code = NotFound desc = could not find container \"29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92\": container with ID starting with 29d14925c9efefd9ea4f6bf259285db858feae7b330f0b200989ee1765769f92 not found: ID does not exist"
Sep 09 21:57:33.317512 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.317446    2579 scope.go:117] "RemoveContainer" containerID="64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03"
Sep 09 21:57:33.328543 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.328523    2579 scope.go:117] "RemoveContainer" containerID="64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03"
Sep 09 21:57:33.328888 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:57:33.328868    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03\": container with ID starting with 64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03 not found: ID does not exist" containerID="64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03"
Sep 09 21:57:33.328948 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.328895    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03"} err="failed to get container status \"64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03\": rpc error: code = NotFound desc = could not find container \"64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03\": container with ID starting with 64bc5641b928a36ea056800cafb7b5667b0edcce1c2807006a16d49dd413ae03 not found: ID does not exist"
Sep 09 21:57:33.336702 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.336674    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"]
Sep 09 21:57:33.340577 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.340551    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5677f8b9-6dfd97754b-d5fhx"]
Sep 09 21:57:33.350458 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.350437    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"]
Sep 09 21:57:33.358870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:33.358849    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5677f8b9-6dc5549ccc-znshv"]
Sep 09 21:57:34.093226 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:34.093190    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bae968-d1ce-4e82-af38-1edd92d906aa" path="/var/lib/kubelet/pods/41bae968-d1ce-4e82-af38-1edd92d906aa/volumes"
Sep 09 21:57:34.093634 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:34.093585    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aae71806-0a4e-46d3-8dec-89eb72721b69" path="/var/lib/kubelet/pods/aae71806-0a4e-46d3-8dec-89eb72721b69/volumes"
Sep 09 21:57:34.093907 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:34.093893    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc3f484-3da7-41af-8a9c-c78fb664041f" path="/var/lib/kubelet/pods/acc3f484-3da7-41af-8a9c-c78fb664041f/volumes"
Sep 09 21:57:55.621681 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:55.621638    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"]
Sep 09 21:57:55.622185 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:55.621913    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r" podUID="a37012b2-4d6f-49ab-948e-26fe35dea7a1" containerName="payload-processing" containerID="cri-o://08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72" gracePeriod=30
Sep 09 21:57:55.630144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:55.629856    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"]
Sep 09 21:57:55.630144 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:55.630088    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z" podUID="63bd80d3-3476-48a0-9140-29ecd5d7ffe0" containerName="payload-pre-processing" containerID="cri-o://c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d" gracePeriod=30
Sep 09 21:57:55.910571 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:55.910543    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:57:55.916501 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:55.916479    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:57:56.075662 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.075593    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-plugins-config-volume\") pod \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\" (UID: \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\") "
Sep 09 21:57:56.075845 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.075725    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37012b2-4d6f-49ab-948e-26fe35dea7a1-plugins-config-volume\") pod \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\" (UID: \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\") "
Sep 09 21:57:56.075845 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.075777    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8gqj\" (UniqueName: \"kubernetes.io/projected/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-kube-api-access-f8gqj\") pod \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\" (UID: \"63bd80d3-3476-48a0-9140-29ecd5d7ffe0\") "
Sep 09 21:57:56.075845 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.075805    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8prf\" (UniqueName: \"kubernetes.io/projected/a37012b2-4d6f-49ab-948e-26fe35dea7a1-kube-api-access-b8prf\") pod \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\" (UID: \"a37012b2-4d6f-49ab-948e-26fe35dea7a1\") "
Sep 09 21:57:56.076024 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.076001    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63bd80d3-3476-48a0-9140-29ecd5d7ffe0" (UID: "63bd80d3-3476-48a0-9140-29ecd5d7ffe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:57:56.076122 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.076091    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37012b2-4d6f-49ab-948e-26fe35dea7a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a37012b2-4d6f-49ab-948e-26fe35dea7a1" (UID: "a37012b2-4d6f-49ab-948e-26fe35dea7a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:57:56.076240 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.076119    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:56.078009 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.077988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37012b2-4d6f-49ab-948e-26fe35dea7a1-kube-api-access-b8prf" (OuterVolumeSpecName: "kube-api-access-b8prf") pod "a37012b2-4d6f-49ab-948e-26fe35dea7a1" (UID: "a37012b2-4d6f-49ab-948e-26fe35dea7a1"). InnerVolumeSpecName "kube-api-access-b8prf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:57:56.078082 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.078017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-kube-api-access-f8gqj" (OuterVolumeSpecName: "kube-api-access-f8gqj") pod "63bd80d3-3476-48a0-9140-29ecd5d7ffe0" (UID: "63bd80d3-3476-48a0-9140-29ecd5d7ffe0"). InnerVolumeSpecName "kube-api-access-f8gqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:57:56.177510 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.177418    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37012b2-4d6f-49ab-948e-26fe35dea7a1-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:56.177510 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.177457    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8gqj\" (UniqueName: \"kubernetes.io/projected/63bd80d3-3476-48a0-9140-29ecd5d7ffe0-kube-api-access-f8gqj\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:56.177510 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.177470    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8prf\" (UniqueName: \"kubernetes.io/projected/a37012b2-4d6f-49ab-948e-26fe35dea7a1-kube-api-access-b8prf\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:57:56.444854 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.444765    2579 generic.go:356] "Generic (PLEG): container finished" podID="63bd80d3-3476-48a0-9140-29ecd5d7ffe0" containerID="c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d" exitCode=0
Sep 09 21:57:56.444854 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.444804    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"
Sep 09 21:57:56.445059 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.444852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z" event={"ID":"63bd80d3-3476-48a0-9140-29ecd5d7ffe0","Type":"ContainerDied","Data":"c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d"}
Sep 09 21:57:56.445059 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.444905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z" event={"ID":"63bd80d3-3476-48a0-9140-29ecd5d7ffe0","Type":"ContainerDied","Data":"73ab8a5a4af7d48d24e9ac4ac65cb76ac75c034fcc75dd7ebf63206c7311e09a"}
Sep 09 21:57:56.445059 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.444934    2579 scope.go:117] "RemoveContainer" containerID="c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d"
Sep 09 21:57:56.446585 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.446566    2579 generic.go:356] "Generic (PLEG): container finished" podID="a37012b2-4d6f-49ab-948e-26fe35dea7a1" containerID="08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72" exitCode=0
Sep 09 21:57:56.446717 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.446624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r" event={"ID":"a37012b2-4d6f-49ab-948e-26fe35dea7a1","Type":"ContainerDied","Data":"08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72"}
Sep 09 21:57:56.446717 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.446632    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"
Sep 09 21:57:56.446717 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.446666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r" event={"ID":"a37012b2-4d6f-49ab-948e-26fe35dea7a1","Type":"ContainerDied","Data":"e8072929408ee3798ad325737e5aa1de643723d1299080b82ad05feccbb4290b"}
Sep 09 21:57:56.454003 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.453990    2579 scope.go:117] "RemoveContainer" containerID="c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d"
Sep 09 21:57:56.454291 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:57:56.454271    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d\": container with ID starting with c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d not found: ID does not exist" containerID="c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d"
Sep 09 21:57:56.454372 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.454297    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d"} err="failed to get container status \"c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d\": rpc error: code = NotFound desc = could not find container \"c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d\": container with ID starting with c80fd001a77ca2b718c9ee34a6f3f34d0d82bf661b2de031475fd05a206f846d not found: ID does not exist"
Sep 09 21:57:56.454372 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.454314    2579 scope.go:117] "RemoveContainer" containerID="08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72"
Sep 09 21:57:56.462264 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.462234    2579 scope.go:117] "RemoveContainer" containerID="08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72"
Sep 09 21:57:56.462548 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:57:56.462526    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72\": container with ID starting with 08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72 not found: ID does not exist" containerID="08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72"
Sep 09 21:57:56.462633 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.462557    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72"} err="failed to get container status \"08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72\": rpc error: code = NotFound desc = could not find container \"08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72\": container with ID starting with 08df0bbb069be449a9c06ffd6367352e18c392b55c5a1e45a8020fdb0cc2dc72 not found: ID does not exist"
Sep 09 21:57:56.493451 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.493414    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"]
Sep 09 21:57:56.495298 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.495279    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbdd85-6448cdcd46-vf67z"]
Sep 09 21:57:56.506809 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.506785    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"]
Sep 09 21:57:56.511963 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:56.511937    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbdd85-7cb9584c7-mbn7r"]
Sep 09 21:57:58.092880 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:58.092843    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63bd80d3-3476-48a0-9140-29ecd5d7ffe0" path="/var/lib/kubelet/pods/63bd80d3-3476-48a0-9140-29ecd5d7ffe0/volumes"
Sep 09 21:57:58.093268 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:57:58.093194    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37012b2-4d6f-49ab-948e-26fe35dea7a1" path="/var/lib/kubelet/pods/a37012b2-4d6f-49ab-948e-26fe35dea7a1/volumes"
Sep 09 21:58:48.992471 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:48.992435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"]
Sep 09 21:58:48.992967 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:48.992695    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk" podUID="c0cf131e-eded-47d8-9f02-02d12d171a91" containerName="payload-processing" containerID="cri-o://3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef" gracePeriod=30
Sep 09 21:58:49.011070 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.011029    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"]
Sep 09 21:58:49.011316 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.011288    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6" podUID="11c26949-7864-4fe7-a707-4fcbf0707078" containerName="payload-pre-processing" containerID="cri-o://0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a" gracePeriod=30
Sep 09 21:58:49.174800 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.174367    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6" podUID="11c26949-7864-4fe7-a707-4fcbf0707078" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.40:9004: connect: connection refused"
Sep 09 21:58:49.273371 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.273348    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:58:49.300087 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.300023    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:58:49.407772 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.407741    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cf131e-eded-47d8-9f02-02d12d171a91-plugins-config-volume\") pod \"c0cf131e-eded-47d8-9f02-02d12d171a91\" (UID: \"c0cf131e-eded-47d8-9f02-02d12d171a91\") "
Sep 09 21:58:49.407976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.407904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c26949-7864-4fe7-a707-4fcbf0707078-plugins-config-volume\") pod \"11c26949-7864-4fe7-a707-4fcbf0707078\" (UID: \"11c26949-7864-4fe7-a707-4fcbf0707078\") "
Sep 09 21:58:49.407976 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.407964    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rdf9\" (UniqueName: \"kubernetes.io/projected/11c26949-7864-4fe7-a707-4fcbf0707078-kube-api-access-5rdf9\") pod \"11c26949-7864-4fe7-a707-4fcbf0707078\" (UID: \"11c26949-7864-4fe7-a707-4fcbf0707078\") "
Sep 09 21:58:49.408092 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.407999    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ktzj\" (UniqueName: \"kubernetes.io/projected/c0cf131e-eded-47d8-9f02-02d12d171a91-kube-api-access-7ktzj\") pod \"c0cf131e-eded-47d8-9f02-02d12d171a91\" (UID: \"c0cf131e-eded-47d8-9f02-02d12d171a91\") "
Sep 09 21:58:49.408143 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.408077    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cf131e-eded-47d8-9f02-02d12d171a91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0cf131e-eded-47d8-9f02-02d12d171a91" (UID: "c0cf131e-eded-47d8-9f02-02d12d171a91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:49.408245 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.408224    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c26949-7864-4fe7-a707-4fcbf0707078-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11c26949-7864-4fe7-a707-4fcbf0707078" (UID: "11c26949-7864-4fe7-a707-4fcbf0707078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:49.408343 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.408326    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cf131e-eded-47d8-9f02-02d12d171a91-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:58:49.408399 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.408349    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c26949-7864-4fe7-a707-4fcbf0707078-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:58:49.410089 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.410064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c26949-7864-4fe7-a707-4fcbf0707078-kube-api-access-5rdf9" (OuterVolumeSpecName: "kube-api-access-5rdf9") pod "11c26949-7864-4fe7-a707-4fcbf0707078" (UID: "11c26949-7864-4fe7-a707-4fcbf0707078"). InnerVolumeSpecName "kube-api-access-5rdf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:49.410200 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.410089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cf131e-eded-47d8-9f02-02d12d171a91-kube-api-access-7ktzj" (OuterVolumeSpecName: "kube-api-access-7ktzj") pod "c0cf131e-eded-47d8-9f02-02d12d171a91" (UID: "c0cf131e-eded-47d8-9f02-02d12d171a91"). InnerVolumeSpecName "kube-api-access-7ktzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:58:49.509337 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.509248    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rdf9\" (UniqueName: \"kubernetes.io/projected/11c26949-7864-4fe7-a707-4fcbf0707078-kube-api-access-5rdf9\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:58:49.509337 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.509279    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktzj\" (UniqueName: \"kubernetes.io/projected/c0cf131e-eded-47d8-9f02-02d12d171a91-kube-api-access-7ktzj\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:58:49.743304 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.743262    2579 generic.go:356] "Generic (PLEG): container finished" podID="11c26949-7864-4fe7-a707-4fcbf0707078" containerID="0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a" exitCode=0
Sep 09 21:58:49.743304 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.743309    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"
Sep 09 21:58:49.743530 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.743351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6" event={"ID":"11c26949-7864-4fe7-a707-4fcbf0707078","Type":"ContainerDied","Data":"0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a"}
Sep 09 21:58:49.743530 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.743412    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6" event={"ID":"11c26949-7864-4fe7-a707-4fcbf0707078","Type":"ContainerDied","Data":"5b9032d8c49a3d23c99104bd1c7dd58bfad41f46cdb79b4b486abf864eb393bf"}
Sep 09 21:58:49.743530 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.743443    2579 scope.go:117] "RemoveContainer" containerID="0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a"
Sep 09 21:58:49.744930 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.744905    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0cf131e-eded-47d8-9f02-02d12d171a91" containerID="3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef" exitCode=0
Sep 09 21:58:49.745061 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.744937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk" event={"ID":"c0cf131e-eded-47d8-9f02-02d12d171a91","Type":"ContainerDied","Data":"3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef"}
Sep 09 21:58:49.745061 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.744959    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"
Sep 09 21:58:49.745061 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.744992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk" event={"ID":"c0cf131e-eded-47d8-9f02-02d12d171a91","Type":"ContainerDied","Data":"372b947ab9f968132cb9e1eec365ca7e6534a300fc519657bf7e6203678e4660"}
Sep 09 21:58:49.754540 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.754519    2579 scope.go:117] "RemoveContainer" containerID="0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a"
Sep 09 21:58:49.754957 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:58:49.754883    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a\": container with ID starting with 0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a not found: ID does not exist" containerID="0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a"
Sep 09 21:58:49.754957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.754912    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a"} err="failed to get container status \"0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a\": rpc error: code = NotFound desc = could not find container \"0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a\": container with ID starting with 0c5b83271727fa206135f3aaf3f8e71f2ef0f8c4de63ae05f3e6f72af731833a not found: ID does not exist"
Sep 09 21:58:49.754957 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.754931    2579 scope.go:117] "RemoveContainer" containerID="3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef"
Sep 09 21:58:49.763255 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.763236    2579 scope.go:117] "RemoveContainer" containerID="3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef"
Sep 09 21:58:49.763548 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:58:49.763529    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef\": container with ID starting with 3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef not found: ID does not exist" containerID="3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef"
Sep 09 21:58:49.763640 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.763554    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef"} err="failed to get container status \"3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef\": rpc error: code = NotFound desc = could not find container \"3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef\": container with ID starting with 3bda7195643898dfa90331632d25a126b6f0d870c9027b1cc817c6f044a849ef not found: ID does not exist"
Sep 09 21:58:49.769046 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.769018    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"]
Sep 09 21:58:49.773447 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.773423    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5aaa80-797f7db946-7b4n6"]
Sep 09 21:58:49.786996 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.786966    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"]
Sep 09 21:58:49.793253 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:49.793226    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk"]
Sep 09 21:58:50.092462 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:50.092375    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c26949-7864-4fe7-a707-4fcbf0707078" path="/var/lib/kubelet/pods/11c26949-7864-4fe7-a707-4fcbf0707078/volumes"
Sep 09 21:58:50.092827 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:50.092778    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0cf131e-eded-47d8-9f02-02d12d171a91" path="/var/lib/kubelet/pods/c0cf131e-eded-47d8-9f02-02d12d171a91/volumes"
Sep 09 21:58:50.174212 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:50.174169    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5aaa80-65567c65-z8xtk" podUID="c0cf131e-eded-47d8-9f02-02d12d171a91" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.41:9004: i/o timeout"
Sep 09 21:58:56.046453 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.046421    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"]
Sep 09 21:58:56.046992 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.046677    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy" containerID="cri-o://95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c" gracePeriod=30
Sep 09 21:58:56.390692 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.390664    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"]
Sep 09 21:58:56.391076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391063    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63bd80d3-3476-48a0-9140-29ecd5d7ffe0" containerName="payload-pre-processing"
Sep 09 21:58:56.391126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391077    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63bd80d3-3476-48a0-9140-29ecd5d7ffe0" containerName="payload-pre-processing"
Sep 09 21:58:56.391126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391084    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37012b2-4d6f-49ab-948e-26fe35dea7a1" containerName="payload-processing"
Sep 09 21:58:56.391126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391090    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37012b2-4d6f-49ab-948e-26fe35dea7a1" containerName="payload-processing"
Sep 09 21:58:56.391126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391104    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c26949-7864-4fe7-a707-4fcbf0707078" containerName="payload-pre-processing"
Sep 09 21:58:56.391126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391110    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c26949-7864-4fe7-a707-4fcbf0707078" containerName="payload-pre-processing"
Sep 09 21:58:56.391126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391118    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc3f484-3da7-41af-8a9c-c78fb664041f" containerName="revoke-keys"
Sep 09 21:58:56.391126 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391123    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc3f484-3da7-41af-8a9c-c78fb664041f" containerName="revoke-keys"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391134    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0cf131e-eded-47d8-9f02-02d12d171a91" containerName="payload-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391140    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0cf131e-eded-47d8-9f02-02d12d171a91" containerName="payload-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391149    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae71806-0a4e-46d3-8dec-89eb72721b69" containerName="payload-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391155    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae71806-0a4e-46d3-8dec-89eb72721b69" containerName="payload-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391164    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bae968-d1ce-4e82-af38-1edd92d906aa" containerName="payload-pre-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391169    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bae968-d1ce-4e82-af38-1edd92d906aa" containerName="payload-pre-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391226    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bae968-d1ce-4e82-af38-1edd92d906aa" containerName="payload-pre-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391233    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c26949-7864-4fe7-a707-4fcbf0707078" containerName="payload-pre-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391239    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aae71806-0a4e-46d3-8dec-89eb72721b69" containerName="payload-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391246    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc3f484-3da7-41af-8a9c-c78fb664041f" containerName="revoke-keys"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391252    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37012b2-4d6f-49ab-948e-26fe35dea7a1" containerName="payload-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391256    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63bd80d3-3476-48a0-9140-29ecd5d7ffe0" containerName="payload-pre-processing"
Sep 09 21:58:56.391361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.391263    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0cf131e-eded-47d8-9f02-02d12d171a91" containerName="payload-processing"
Sep 09 21:58:56.395682 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.395665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"
Sep 09 21:58:56.398290 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.398269    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 21:58:56.403479 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.402988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"]
Sep 09 21:58:56.475268 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.475234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd\" (UID: \"f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"
Sep 09 21:58:56.576580 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.576537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd\" (UID: \"f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"
Sep 09 21:58:56.577251 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.577228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd\" (UID: \"f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"
Sep 09 21:58:56.707863 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:56.707771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"
Sep 09 21:58:57.001704 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:57.001676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"]
Sep 09 21:58:57.004445 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:58:57.004413    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e84e39_fd2e_4fdc_bf78_fa32d7a03bb5.slice/crio-4b56079dd4610f5fe8440fe0c55514ee26ce9da5b2a1410ad3ded992c077c97d WatchSource:0}: Error finding container 4b56079dd4610f5fe8440fe0c55514ee26ce9da5b2a1410ad3ded992c077c97d: Status 404 returned error can't find the container with id 4b56079dd4610f5fe8440fe0c55514ee26ce9da5b2a1410ad3ded992c077c97d
Sep 09 21:58:57.792907 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:57.792870    2579 generic.go:356] "Generic (PLEG): container finished" podID="f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5" containerID="01f75bcffb7e83f4dbd47ec6c5df2293c481a7a5529477cc9b98e1d51fe23000" exitCode=0
Sep 09 21:58:57.793317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:57.792931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd" event={"ID":"f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5","Type":"ContainerDied","Data":"01f75bcffb7e83f4dbd47ec6c5df2293c481a7a5529477cc9b98e1d51fe23000"}
Sep 09 21:58:57.793317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:57.793014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd" event={"ID":"f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5","Type":"ContainerStarted","Data":"4b56079dd4610f5fe8440fe0c55514ee26ce9da5b2a1410ad3ded992c077c97d"}
Sep 09 21:58:58.940895 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:58.940871    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"
Sep 09 21:58:59.099834 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:59.099808    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5-maas-api-service-ca\") pod \"f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5\" (UID: \"f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5\") "
Sep 09 21:58:59.100196 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:59.100170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5" (UID: "f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:58:59.200921 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:59.200883    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:58:59.804668 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:59.804634    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"
Sep 09 21:58:59.804668 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:59.804647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd" event={"ID":"f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5","Type":"ContainerDied","Data":"4b56079dd4610f5fe8440fe0c55514ee26ce9da5b2a1410ad3ded992c077c97d"}
Sep 09 21:58:59.804875 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:58:59.804699    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b56079dd4610f5fe8440fe0c55514ee26ce9da5b2a1410ad3ded992c077c97d"
Sep 09 21:59:01.293225 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.293201    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:59:01.420152 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420113    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-data\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.420406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-istio-token\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.420406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4k6z\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-kube-api-access-m4k6z\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.420406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-credential-socket\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.420406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-socket\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.420657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:01.420657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420434    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-envoy\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.420657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-certs\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.420657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420532    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:01.420657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133a7cdf-2548-4f74-979c-633d884a16ef-istiod-ca-cert\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.420657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420576    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:01.420657 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133a7cdf-2548-4f74-979c-633d884a16ef-istio-podinfo\") pod \"133a7cdf-2548-4f74-979c-633d884a16ef\" (UID: \"133a7cdf-2548-4f74-979c-633d884a16ef\") "
Sep 09 21:59:01.421013 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:01.421013 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.420976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133a7cdf-2548-4f74-979c-633d884a16ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:01.421104 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.421094    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.421141 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.421110    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.421141 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.421125    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.421206 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.421138    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133a7cdf-2548-4f74-979c-633d884a16ef-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.421206 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.421152    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.422521 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.422499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:01.422844 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.422821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/133a7cdf-2548-4f74-979c-633d884a16ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:59:01.422966 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.422947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:01.423011 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.422983    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-kube-api-access-m4k6z" (OuterVolumeSpecName: "kube-api-access-m4k6z") pod "133a7cdf-2548-4f74-979c-633d884a16ef" (UID: "133a7cdf-2548-4f74-979c-633d884a16ef"). InnerVolumeSpecName "kube-api-access-m4k6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:01.522624 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.522572    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.522624 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.522598    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4k6z\" (UniqueName: \"kubernetes.io/projected/133a7cdf-2548-4f74-979c-633d884a16ef-kube-api-access-m4k6z\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.522825 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.522637    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133a7cdf-2548-4f74-979c-633d884a16ef-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.522825 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.522651    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133a7cdf-2548-4f74-979c-633d884a16ef-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:01.818273 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.818173    2579 generic.go:356] "Generic (PLEG): container finished" podID="133a7cdf-2548-4f74-979c-633d884a16ef" containerID="95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c" exitCode=0
Sep 09 21:59:01.818273 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.818245    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"
Sep 09 21:59:01.818475 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.818268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" event={"ID":"133a7cdf-2548-4f74-979c-633d884a16ef","Type":"ContainerDied","Data":"95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c"}
Sep 09 21:59:01.818475 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.818345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728" event={"ID":"133a7cdf-2548-4f74-979c-633d884a16ef","Type":"ContainerDied","Data":"11fe2cfc052c70438b98899b46096088c0bffc032f111dbbacd2a0f8a487964b"}
Sep 09 21:59:01.818475 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.818377    2579 scope.go:117] "RemoveContainer" containerID="95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c"
Sep 09 21:59:01.828012 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.827995    2579 scope.go:117] "RemoveContainer" containerID="668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390"
Sep 09 21:59:01.836031 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.836015    2579 scope.go:117] "RemoveContainer" containerID="95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c"
Sep 09 21:59:01.836320 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:59:01.836302    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c\": container with ID starting with 95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c not found: ID does not exist" containerID="95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c"
Sep 09 21:59:01.836369 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.836329    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c"} err="failed to get container status \"95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c\": rpc error: code = NotFound desc = could not find container \"95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c\": container with ID starting with 95b9dd6b786e3dbd7499ff335f86844bba738525efe677cf6ec92b97d6672f4c not found: ID does not exist"
Sep 09 21:59:01.836369 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.836346    2579 scope.go:117] "RemoveContainer" containerID="668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390"
Sep 09 21:59:01.836591 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:59:01.836573    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390\": container with ID starting with 668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390 not found: ID does not exist" containerID="668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390"
Sep 09 21:59:01.836673 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.836615    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390"} err="failed to get container status \"668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390\": rpc error: code = NotFound desc = could not find container \"668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390\": container with ID starting with 668e7117624542431c12f5f8c2ab92f29ea5d55d17e0ce7f67d596ad5f74c390 not found: ID does not exist"
Sep 09 21:59:01.850659 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.850628    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"]
Sep 09 21:59:01.858669 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:01.858646    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5aaa80-openshift-default-69fb798dd6-hr728"]
Sep 09 21:59:02.092633 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:02.092530    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" path="/var/lib/kubelet/pods/133a7cdf-2548-4f74-979c-633d884a16ef/volumes"
Sep 09 21:59:07.312808 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.312769    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"]
Sep 09 21:59:07.313295 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.313028    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d" podUID="8a12e563-fb9c-4fdf-9bef-ee851285bb73" containerName="payload-processing" containerID="cri-o://1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826" gracePeriod=30
Sep 09 21:59:07.563293 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.563234    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:59:07.619552 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.619512    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"]
Sep 09 21:59:07.624617 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.624571    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ebe9b0-z7jzd"]
Sep 09 21:59:07.684014 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.683983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv962\" (UniqueName: \"kubernetes.io/projected/8a12e563-fb9c-4fdf-9bef-ee851285bb73-kube-api-access-rv962\") pod \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\" (UID: \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\") "
Sep 09 21:59:07.684179 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.684086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a12e563-fb9c-4fdf-9bef-ee851285bb73-plugins-config-volume\") pod \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\" (UID: \"8a12e563-fb9c-4fdf-9bef-ee851285bb73\") "
Sep 09 21:59:07.684451 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.684430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a12e563-fb9c-4fdf-9bef-ee851285bb73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a12e563-fb9c-4fdf-9bef-ee851285bb73" (UID: "8a12e563-fb9c-4fdf-9bef-ee851285bb73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:07.686077 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.686051    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a12e563-fb9c-4fdf-9bef-ee851285bb73-kube-api-access-rv962" (OuterVolumeSpecName: "kube-api-access-rv962") pod "8a12e563-fb9c-4fdf-9bef-ee851285bb73" (UID: "8a12e563-fb9c-4fdf-9bef-ee851285bb73"). InnerVolumeSpecName "kube-api-access-rv962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:07.785269 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.785234    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv962\" (UniqueName: \"kubernetes.io/projected/8a12e563-fb9c-4fdf-9bef-ee851285bb73-kube-api-access-rv962\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:07.785269 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.785266    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a12e563-fb9c-4fdf-9bef-ee851285bb73-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:07.851963 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.851931    2579 generic.go:356] "Generic (PLEG): container finished" podID="8a12e563-fb9c-4fdf-9bef-ee851285bb73" containerID="1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826" exitCode=0
Sep 09 21:59:07.852107 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.851971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"
Sep 09 21:59:07.852107 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.852011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d" event={"ID":"8a12e563-fb9c-4fdf-9bef-ee851285bb73","Type":"ContainerDied","Data":"1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826"}
Sep 09 21:59:07.852107 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.852064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d" event={"ID":"8a12e563-fb9c-4fdf-9bef-ee851285bb73","Type":"ContainerDied","Data":"2ec657c732857d27a98abd9e76c63402989a51085008204a3f333fb9f76a8dee"}
Sep 09 21:59:07.852107 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.852086    2579 scope.go:117] "RemoveContainer" containerID="1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826"
Sep 09 21:59:07.861519 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.861504    2579 scope.go:117] "RemoveContainer" containerID="1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826"
Sep 09 21:59:07.861791 ip-10-0-139-221 kubenswrapper[2579]: E0909 21:59:07.861773    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826\": container with ID starting with 1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826 not found: ID does not exist" containerID="1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826"
Sep 09 21:59:07.861886 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.861798    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826"} err="failed to get container status \"1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826\": rpc error: code = NotFound desc = could not find container \"1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826\": container with ID starting with 1643aa6461831a0f41574253067644adabfbe24b89b44a0c339b8009c09a5826 not found: ID does not exist"
Sep 09 21:59:07.878391 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.878365    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"]
Sep 09 21:59:07.884762 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:07.884737    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ebe9b0-fbfb595b5-qtz2d"]
Sep 09 21:59:08.092440 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:08.092405    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a12e563-fb9c-4fdf-9bef-ee851285bb73" path="/var/lib/kubelet/pods/8a12e563-fb9c-4fdf-9bef-ee851285bb73/volumes"
Sep 09 21:59:08.092812 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:08.092797    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5" path="/var/lib/kubelet/pods/f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5/volumes"
Sep 09 21:59:13.552261 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552227    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"]
Sep 09 21:59:13.552767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552636    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5" containerName="revoke-keys"
Sep 09 21:59:13.552767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552649    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5" containerName="revoke-keys"
Sep 09 21:59:13.552767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552659    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy"
Sep 09 21:59:13.552767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552665    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy"
Sep 09 21:59:13.552767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552685    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a12e563-fb9c-4fdf-9bef-ee851285bb73" containerName="payload-processing"
Sep 09 21:59:13.552767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552690    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a12e563-fb9c-4fdf-9bef-ee851285bb73" containerName="payload-processing"
Sep 09 21:59:13.552767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552764    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy"
Sep 09 21:59:13.552767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552772    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a12e563-fb9c-4fdf-9bef-ee851285bb73" containerName="payload-processing"
Sep 09 21:59:13.553038 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.552778    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e84e39-fd2e-4fdc-bf78-fa32d7a03bb5" containerName="revoke-keys"
Sep 09 21:59:13.561474 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.561453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.564113 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.564091    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df8f95ff-openshift-default-dockercfg-c6s2v\""
Sep 09 21:59:13.570765 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.570741    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"]
Sep 09 21:59:13.639395 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-token\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.639395 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-envoy\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.639636 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-socket\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.639636 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b4beaf-7735-4794-9c0d-298ca224bf4a-istiod-ca-cert\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.639636 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfnd\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-kube-api-access-dkfnd\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.639636 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-certs\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.639636 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-podinfo\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.639804 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-credential-socket\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.639804 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.639685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-data\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.740913 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.740876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-token\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.740921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-envoy\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.740940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-socket\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.740961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b4beaf-7735-4794-9c0d-298ca224bf4a-istiod-ca-cert\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.740991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfnd\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-kube-api-access-dkfnd\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-certs\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741076 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-podinfo\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741468 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-credential-socket\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741468 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-data\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741575 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-certs\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741575 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-socket\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741760 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-data\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741822 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-credential-socket\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.741885 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.741791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b4beaf-7735-4794-9c0d-298ca224bf4a-istiod-ca-cert\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.743288 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.743263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-envoy\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.743560 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.743540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-podinfo\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.757642 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.757595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-token\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.757767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.757752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfnd\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-kube-api-access-dkfnd\") pod \"e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") " pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:13.872651 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:13.872618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:14.165414 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:14.165388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"]
Sep 09 21:59:14.166963 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:59:14.166934    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b4beaf_7735_4794_9c0d_298ca224bf4a.slice/crio-b782ffa82ba621b21606e958bbf5c1c1c08933fc16b444aaa0d326115e34f8cf WatchSource:0}: Error finding container b782ffa82ba621b21606e958bbf5c1c1c08933fc16b444aaa0d326115e34f8cf: Status 404 returned error can't find the container with id b782ffa82ba621b21606e958bbf5c1c1c08933fc16b444aaa0d326115e34f8cf
Sep 09 21:59:14.169165 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:14.169135    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:14.169234 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:14.169196    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:14.169234 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:14.169226    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:14.892331 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:14.892295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" event={"ID":"73b4beaf-7735-4794-9c0d-298ca224bf4a","Type":"ContainerStarted","Data":"1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93"}
Sep 09 21:59:14.892750 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:14.892381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" event={"ID":"73b4beaf-7735-4794-9c0d-298ca224bf4a","Type":"ContainerStarted","Data":"b782ffa82ba621b21606e958bbf5c1c1c08933fc16b444aaa0d326115e34f8cf"}
Sep 09 21:59:14.927387 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:14.927336    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podStartSLOduration=1.927319531 podStartE2EDuration="1.927319531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:14.9249273 +0000 UTC m=+1229.347290013" watchObservedRunningTime="2026-09-09 21:59:14.927319531 +0000 UTC m=+1229.349682231"
Sep 09 21:59:15.873132 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:15.873084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:15.878313 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:15.878280    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:15.898017 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:15.897990    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:15.899094 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:15.899072    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:16.991937 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:16.991901    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"]
Sep 09 21:59:16.992324 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:16.992312    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy"
Sep 09 21:59:16.992370 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:16.992326    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy"
Sep 09 21:59:16.992406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:16.992391    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="133a7cdf-2548-4f74-979c-633d884a16ef" containerName="istio-proxy"
Sep 09 21:59:16.995551 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:16.995528    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:16.998760 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:16.998729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ace629-openshift-default-dockercfg-7qckz\""
Sep 09 21:59:17.024689 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.024659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"]
Sep 09 21:59:17.078073 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istiod-ca-cert\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.078249 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-credential-socket\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.078249 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-socket\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.078249 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-token\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.078249 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-podinfo\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.078450 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-certs\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.078450 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-envoy\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.078450 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-data\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.078450 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.078399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhgkx\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-kube-api-access-vhgkx\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.179815 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.179764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-credential-socket\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.179835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-socket\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.179863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-token\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.179900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-podinfo\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.179963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-certs\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180032 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.179987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-envoy\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.180057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-data\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.180166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhgkx\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-kube-api-access-vhgkx\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.180268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-credential-socket\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180317 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.180301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istiod-ca-cert\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180561 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.180310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-socket\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180719 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.180697    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-data\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.180870 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.180850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-certs\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.181081 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.181061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istiod-ca-cert\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.182650 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.182625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-envoy\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.182763 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.182631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-podinfo\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.196767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.196729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-token\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.196936 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.196822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhgkx\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-kube-api-access-vhgkx\") pod \"e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") " pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.307322 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.307225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:17.589412 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.589381    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"]
Sep 09 21:59:17.591128 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:59:17.591101    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed05a1f_0e85_48ec_8c0d_dbc9159e86b7.slice/crio-d2bae3c994686f0b52cca6f77a03d2a5064f876e0d210e3d1426f6a8dc7d3595 WatchSource:0}: Error finding container d2bae3c994686f0b52cca6f77a03d2a5064f876e0d210e3d1426f6a8dc7d3595: Status 404 returned error can't find the container with id d2bae3c994686f0b52cca6f77a03d2a5064f876e0d210e3d1426f6a8dc7d3595
Sep 09 21:59:17.593531 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.593491    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:17.593635 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.593583    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:17.593685 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.593645    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:17.913141 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.913106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" event={"ID":"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7","Type":"ContainerStarted","Data":"be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa"}
Sep 09 21:59:17.913361 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.913167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" event={"ID":"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7","Type":"ContainerStarted","Data":"d2bae3c994686f0b52cca6f77a03d2a5064f876e0d210e3d1426f6a8dc7d3595"}
Sep 09 21:59:17.942980 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:17.942912    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podStartSLOduration=1.942889973 podStartE2EDuration="1.942889973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:17.938953309 +0000 UTC m=+1232.361316013" watchObservedRunningTime="2026-09-09 21:59:17.942889973 +0000 UTC m=+1232.365252674"
Sep 09 21:59:18.308164 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:18.308071    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:18.309699 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:18.309667    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 21:59:18.309844 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:18.309728    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 21:59:18.933483 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:18.933449    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"]
Sep 09 21:59:18.947168 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:18.947141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:18.950406 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:18.950378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df8f95ff-dockercfg-tfzn8\""
Sep 09 21:59:18.950542 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:18.950379    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df8f95ff\""
Sep 09 21:59:18.959713 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:18.959691    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"]
Sep 09 21:59:19.001203 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.001158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8j5s\" (UniqueName: \"kubernetes.io/projected/9c1edf93-16ad-4518-b3ef-84a06769fe5f-kube-api-access-k8j5s\") pod \"payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj\" (UID: \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:19.001385 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.001230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1edf93-16ad-4518-b3ef-84a06769fe5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj\" (UID: \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:19.102010 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.101945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8j5s\" (UniqueName: \"kubernetes.io/projected/9c1edf93-16ad-4518-b3ef-84a06769fe5f-kube-api-access-k8j5s\") pod \"payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj\" (UID: \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:19.102165 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.102039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1edf93-16ad-4518-b3ef-84a06769fe5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj\" (UID: \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:19.103058 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.103031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1edf93-16ad-4518-b3ef-84a06769fe5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj\" (UID: \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:19.112377 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.112335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8j5s\" (UniqueName: \"kubernetes.io/projected/9c1edf93-16ad-4518-b3ef-84a06769fe5f-kube-api-access-k8j5s\") pod \"payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj\" (UID: \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:19.260299 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.260202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:19.313220 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.313194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:19.535886 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.535858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"]
Sep 09 21:59:19.537709 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:59:19.537680    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1edf93_16ad_4518_b3ef_84a06769fe5f.slice/crio-a31f3d305edb9b8acede91807e0689534628fe34c41261b51c602dbdff5281a5 WatchSource:0}: Error finding container a31f3d305edb9b8acede91807e0689534628fe34c41261b51c602dbdff5281a5: Status 404 returned error can't find the container with id a31f3d305edb9b8acede91807e0689534628fe34c41261b51c602dbdff5281a5
Sep 09 21:59:19.925149 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.925114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj" event={"ID":"9c1edf93-16ad-4518-b3ef-84a06769fe5f","Type":"ContainerStarted","Data":"4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25"}
Sep 09 21:59:19.925349 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.925204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:19.925432 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.925385    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:19.925500 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.925428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj" event={"ID":"9c1edf93-16ad-4518-b3ef-84a06769fe5f","Type":"ContainerStarted","Data":"a31f3d305edb9b8acede91807e0689534628fe34c41261b51c602dbdff5281a5"}
Sep 09 21:59:19.926386 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.926370    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:19.967360 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:19.967307    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj" podStartSLOduration=1.9672922769999999 podStartE2EDuration="1.967292277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:19.964308675 +0000 UTC m=+1234.386671374" watchObservedRunningTime="2026-09-09 21:59:19.967292277 +0000 UTC m=+1234.389654977"
Sep 09 21:59:22.602355 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.602322    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"]
Sep 09 21:59:22.606062 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.606042    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:22.609767 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.609748    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ace629\""
Sep 09 21:59:22.609987 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.609972    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ace629-dockercfg-wjft7\""
Sep 09 21:59:22.622235 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.622212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"]
Sep 09 21:59:22.739627 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.739566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809528d-8b1c-4e09-86e7-d1fef6d9f559-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6\" (UID: \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:22.739796 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.739755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vj79\" (UniqueName: \"kubernetes.io/projected/9809528d-8b1c-4e09-86e7-d1fef6d9f559-kube-api-access-6vj79\") pod \"payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6\" (UID: \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:22.840726 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.840688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vj79\" (UniqueName: \"kubernetes.io/projected/9809528d-8b1c-4e09-86e7-d1fef6d9f559-kube-api-access-6vj79\") pod \"payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6\" (UID: \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:22.840900 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.840782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809528d-8b1c-4e09-86e7-d1fef6d9f559-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6\" (UID: \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:22.841471 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.841450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809528d-8b1c-4e09-86e7-d1fef6d9f559-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6\" (UID: \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:22.852881 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.852815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vj79\" (UniqueName: \"kubernetes.io/projected/9809528d-8b1c-4e09-86e7-d1fef6d9f559-kube-api-access-6vj79\") pod \"payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6\" (UID: \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:22.916081 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:22.916049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:23.189274 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:23.189238    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"]
Sep 09 21:59:23.191065 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:59:23.191026    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9809528d_8b1c_4e09_86e7_d1fef6d9f559.slice/crio-b1ea6a38328d1917fbf594813324267bef6786478c75b27a1ba06b774c4bbd92 WatchSource:0}: Error finding container b1ea6a38328d1917fbf594813324267bef6786478c75b27a1ba06b774c4bbd92: Status 404 returned error can't find the container with id b1ea6a38328d1917fbf594813324267bef6786478c75b27a1ba06b774c4bbd92
Sep 09 21:59:23.960200 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:23.960160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6" event={"ID":"9809528d-8b1c-4e09-86e7-d1fef6d9f559","Type":"ContainerStarted","Data":"6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc"}
Sep 09 21:59:23.960561 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:23.960217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6" event={"ID":"9809528d-8b1c-4e09-86e7-d1fef6d9f559","Type":"ContainerStarted","Data":"b1ea6a38328d1917fbf594813324267bef6786478c75b27a1ba06b774c4bbd92"}
Sep 09 21:59:23.960561 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:23.960306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:23.980046 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:23.979999    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6" podStartSLOduration=1.979983495 podStartE2EDuration="1.979983495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:23.978538075 +0000 UTC m=+1238.400900776" watchObservedRunningTime="2026-09-09 21:59:23.979983495 +0000 UTC m=+1238.402346195"
Sep 09 21:59:30.933772 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:30.933733    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 21:59:31.938809 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:31.938769    2579 patch_prober.go:28] interesting pod/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:59:31.939358 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:31.939327    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:59:33.047193 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:33.047160    2579 generic.go:356] "Generic (PLEG): container finished" podID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerID="1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93" exitCode=0
Sep 09 21:59:33.047635 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:33.047232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" event={"ID":"73b4beaf-7735-4794-9c0d-298ca224bf4a","Type":"ContainerDied","Data":"1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93"}
Sep 09 21:59:33.047718 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:33.047701    2579 scope.go:117] "RemoveContainer" containerID="1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93"
Sep 09 21:59:33.048798 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:33.048768    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:33.048869 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:33.048845    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:33.048906 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:33.048879    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:34.060440 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:34.060338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" event={"ID":"73b4beaf-7735-4794-9c0d-298ca224bf4a","Type":"ContainerStarted","Data":"c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea"}
Sep 09 21:59:34.873364 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:34.873321    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:34.873652 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:34.873626    2579 patch_prober.go:28] interesting pod/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 21:59:34.873723 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:34.873689    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 21:59:34.968229 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:34.968195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 21:59:35.873177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:35.873140    2579 patch_prober.go:28] interesting pod/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 21:59:35.873575 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:35.873208    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 21:59:35.966794 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:35.966753    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:59:35.966968 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:35.966852    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:59:36.874048 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:36.874004    2579 patch_prober.go:28] interesting pod/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 21:59:36.874518 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:36.874073    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 21:59:37.873239 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:37.873204    2579 patch_prober.go:28] interesting pod/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 21:59:37.873454 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:37.873268    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 21:59:38.085288 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:38.085257    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerID="be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa" exitCode=0
Sep 09 21:59:38.085288 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:38.085293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" event={"ID":"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7","Type":"ContainerDied","Data":"be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa"}
Sep 09 21:59:38.085844 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:38.085825    2579 scope.go:117] "RemoveContainer" containerID="be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa"
Sep 09 21:59:38.086942 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:38.086917    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:38.087011 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:38.086981    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:38.087047 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:38.087016    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:38.873211 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:38.873172    2579 patch_prober.go:28] interesting pod/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 21:59:38.873419 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:38.873234    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 21:59:39.093493 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:39.093447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" event={"ID":"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7","Type":"ContainerStarted","Data":"02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299"}
Sep 09 21:59:39.308047 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:39.307947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:39.308332 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:39.307996    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 21:59:39.308512 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:39.308487    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 21:59:39.876556 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:39.876525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:39.876935 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:39.876898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:39.877703 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:39.877685    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 21:59:40.308112 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:40.308004    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 21:59:40.308112 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:40.308077    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 21:59:41.308659 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:41.308618    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 21:59:41.309287 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:41.309253    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 21:59:42.307992 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:42.307945    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 21:59:42.308218 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:42.308015    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 21:59:43.316627 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:43.312013    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 21:59:43.316627 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:43.312095    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 21:59:44.403708 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:44.403674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:44.404502 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:44.404474    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:44.405362 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:44.405344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 21:59:45.577293 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.577255    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"]
Sep 09 21:59:45.580104 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.580081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.583423 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.583401    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3b2190-openshift-default-dockercfg-ztsx4\""
Sep 09 21:59:45.600200 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.600169    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"]
Sep 09 21:59:45.612177 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d29cb97-241e-492f-92fb-d7702847c2f6-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.612349 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gqs\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-kube-api-access-j4gqs\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.612422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-data\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.612422 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-certs\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.612503 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-token\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.612503 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-podinfo\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.612569 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-socket\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.612569 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-credential-socket\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.612709 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.612579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-envoy\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714257 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d29cb97-241e-492f-92fb-d7702847c2f6-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gqs\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-kube-api-access-j4gqs\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-data\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-certs\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-token\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714446 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-podinfo\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714737 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-socket\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714737 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-credential-socket\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.714853 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-envoy\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.715019 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.714991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-certs\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.715100 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.715033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-credential-socket\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.715100 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.715072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-data\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.715184 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.715128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-socket\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.715184 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.715148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d29cb97-241e-492f-92fb-d7702847c2f6-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.717048 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.717026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-envoy\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.717291 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.717266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-podinfo\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.730315 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.730292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-token\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.730985 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.730968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gqs\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-kube-api-access-j4gqs\") pod \"e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") " pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:45.890300 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:45.890264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:46.172884 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:46.172854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"]
Sep 09 21:59:46.175399 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:59:46.175370    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d29cb97_241e_492f_92fb_d7702847c2f6.slice/crio-4a992b0460e86b342f041d9960aa32fb43657602a4c738b1c6aa658c7b80dd28 WatchSource:0}: Error finding container 4a992b0460e86b342f041d9960aa32fb43657602a4c738b1c6aa658c7b80dd28: Status 404 returned error can't find the container with id 4a992b0460e86b342f041d9960aa32fb43657602a4c738b1c6aa658c7b80dd28
Sep 09 21:59:46.177865 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:46.177833    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:46.177949 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:46.177930    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:46.177987 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:46.177972    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 21:59:47.157802 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:47.157769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" event={"ID":"2d29cb97-241e-492f-92fb-d7702847c2f6","Type":"ContainerStarted","Data":"7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d"}
Sep 09 21:59:47.158199 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:47.157870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" event={"ID":"2d29cb97-241e-492f-92fb-d7702847c2f6","Type":"ContainerStarted","Data":"4a992b0460e86b342f041d9960aa32fb43657602a4c738b1c6aa658c7b80dd28"}
Sep 09 21:59:47.202544 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:47.202479    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" podStartSLOduration=2.202456891 podStartE2EDuration="2.202456891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:47.199323445 +0000 UTC m=+1261.621686146" watchObservedRunningTime="2026-09-09 21:59:47.202456891 +0000 UTC m=+1261.624819593"
Sep 09 21:59:47.891348 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:47.891304    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:47.896505 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:47.896482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:48.163934 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:48.163832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:48.164950 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:48.164926    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 21:59:49.757526 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:49.757492    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"]
Sep 09 21:59:49.760838 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:49.760809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"
Sep 09 21:59:49.763475 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:49.763456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 21:59:49.769019 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:49.768727    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"]
Sep 09 21:59:49.861347 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:49.861312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0543f76-c9cd-4c05-8a3a-9b472a756381-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c\" (UID: \"e0543f76-c9cd-4c05-8a3a-9b472a756381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"
Sep 09 21:59:49.962090 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:49.962048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0543f76-c9cd-4c05-8a3a-9b472a756381-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c\" (UID: \"e0543f76-c9cd-4c05-8a3a-9b472a756381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"
Sep 09 21:59:49.962703 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:49.962684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0543f76-c9cd-4c05-8a3a-9b472a756381-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c\" (UID: \"e0543f76-c9cd-4c05-8a3a-9b472a756381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"
Sep 09 21:59:50.072855 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:50.072761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"
Sep 09 21:59:50.351485 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:50.351462    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"]
Sep 09 21:59:50.353228 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:59:50.353203    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0543f76_c9cd_4c05_8a3a_9b472a756381.slice/crio-e7ebf9657887e37698d26783f27dc7b259112c8ab5566274e3ecf937ee7fd383 WatchSource:0}: Error finding container e7ebf9657887e37698d26783f27dc7b259112c8ab5566274e3ecf937ee7fd383: Status 404 returned error can't find the container with id e7ebf9657887e37698d26783f27dc7b259112c8ab5566274e3ecf937ee7fd383
Sep 09 21:59:51.183248 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.183157    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0543f76-c9cd-4c05-8a3a-9b472a756381" containerID="291a8fb78db0a0ba42f8eb1f55ff1e39d4c36b16b0967b1e04b82aff4579ef43" exitCode=0
Sep 09 21:59:51.183248 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.183234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c" event={"ID":"e0543f76-c9cd-4c05-8a3a-9b472a756381","Type":"ContainerDied","Data":"291a8fb78db0a0ba42f8eb1f55ff1e39d4c36b16b0967b1e04b82aff4579ef43"}
Sep 09 21:59:51.183770 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.183284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c" event={"ID":"e0543f76-c9cd-4c05-8a3a-9b472a756381","Type":"ContainerStarted","Data":"e7ebf9657887e37698d26783f27dc7b259112c8ab5566274e3ecf937ee7fd383"}
Sep 09 21:59:51.396715 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.396681    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"]
Sep 09 21:59:51.399286 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.399252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:51.401937 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.401913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3b2190-dockercfg-s4st4\""
Sep 09 21:59:51.402143 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.402123    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3b2190\""
Sep 09 21:59:51.415915 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.415887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"]
Sep 09 21:59:51.479101 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.478986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24bf8b2-a02f-4148-b098-2b820fed9147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq\" (UID: \"a24bf8b2-a02f-4148-b098-2b820fed9147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:51.479302 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.479146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47gj\" (UniqueName: \"kubernetes.io/projected/a24bf8b2-a02f-4148-b098-2b820fed9147-kube-api-access-w47gj\") pod \"payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq\" (UID: \"a24bf8b2-a02f-4148-b098-2b820fed9147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:51.580698 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.580654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24bf8b2-a02f-4148-b098-2b820fed9147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq\" (UID: \"a24bf8b2-a02f-4148-b098-2b820fed9147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:51.580869 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.580762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47gj\" (UniqueName: \"kubernetes.io/projected/a24bf8b2-a02f-4148-b098-2b820fed9147-kube-api-access-w47gj\") pod \"payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq\" (UID: \"a24bf8b2-a02f-4148-b098-2b820fed9147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:51.581359 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.581336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24bf8b2-a02f-4148-b098-2b820fed9147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq\" (UID: \"a24bf8b2-a02f-4148-b098-2b820fed9147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:51.590407 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.590380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47gj\" (UniqueName: \"kubernetes.io/projected/a24bf8b2-a02f-4148-b098-2b820fed9147-kube-api-access-w47gj\") pod \"payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq\" (UID: \"a24bf8b2-a02f-4148-b098-2b820fed9147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:51.710087 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:51.710045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:52.035512 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.035487    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"]
Sep 09 21:59:52.037510 ip-10-0-139-221 kubenswrapper[2579]: W0909 21:59:52.037483    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24bf8b2_a02f_4148_b098_2b820fed9147.slice/crio-77fe046ac5dba8dccb07bcac5e765e06001160fb667f8bacef1cf95c15750697 WatchSource:0}: Error finding container 77fe046ac5dba8dccb07bcac5e765e06001160fb667f8bacef1cf95c15750697: Status 404 returned error can't find the container with id 77fe046ac5dba8dccb07bcac5e765e06001160fb667f8bacef1cf95c15750697
Sep 09 21:59:52.190618 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.190567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq" event={"ID":"a24bf8b2-a02f-4148-b098-2b820fed9147","Type":"ContainerStarted","Data":"84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c"}
Sep 09 21:59:52.191088 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.190791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 21:59:52.191088 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.190837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq" event={"ID":"a24bf8b2-a02f-4148-b098-2b820fed9147","Type":"ContainerStarted","Data":"77fe046ac5dba8dccb07bcac5e765e06001160fb667f8bacef1cf95c15750697"}
Sep 09 21:59:52.229545 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.229485    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq" podStartSLOduration=1.229469147 podStartE2EDuration="1.229469147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:59:52.229379284 +0000 UTC m=+1266.651741985" watchObservedRunningTime="2026-09-09 21:59:52.229469147 +0000 UTC m=+1266.651831847"
Sep 09 21:59:52.316508 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.316487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"
Sep 09 21:59:52.388515 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.388481    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0543f76-c9cd-4c05-8a3a-9b472a756381-maas-api-service-ca\") pod \"e0543f76-c9cd-4c05-8a3a-9b472a756381\" (UID: \"e0543f76-c9cd-4c05-8a3a-9b472a756381\") "
Sep 09 21:59:52.388877 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.388857    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0543f76-c9cd-4c05-8a3a-9b472a756381-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0543f76-c9cd-4c05-8a3a-9b472a756381" (UID: "e0543f76-c9cd-4c05-8a3a-9b472a756381"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:52.490056 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:52.490014    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0543f76-c9cd-4c05-8a3a-9b472a756381-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 21:59:53.197523 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:53.197493    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"
Sep 09 21:59:53.198031 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:53.197519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c" event={"ID":"e0543f76-c9cd-4c05-8a3a-9b472a756381","Type":"ContainerDied","Data":"e7ebf9657887e37698d26783f27dc7b259112c8ab5566274e3ecf937ee7fd383"}
Sep 09 21:59:53.198031 ip-10-0-139-221 kubenswrapper[2579]: I0909 21:59:53.197582    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7ebf9657887e37698d26783f27dc7b259112c8ab5566274e3ecf937ee7fd383"
Sep 09 22:00:00.149116 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.149076    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"]
Sep 09 22:00:00.149771 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.149750    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0543f76-c9cd-4c05-8a3a-9b472a756381" containerName="revoke-keys"
Sep 09 22:00:00.149875 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.149774    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0543f76-c9cd-4c05-8a3a-9b472a756381" containerName="revoke-keys"
Sep 09 22:00:00.149948 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.149935    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0543f76-c9cd-4c05-8a3a-9b472a756381" containerName="revoke-keys"
Sep 09 22:00:00.153895 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.153873    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"]
Sep 09 22:00:00.154024 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.153984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"
Sep 09 22:00:00.156793 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.156773    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:00:00.157342 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.157325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"
Sep 09 22:00:00.166994 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.166968    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq"]
Sep 09 22:00:00.167677 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.167655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"
Sep 09 22:00:00.171764 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.171156    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"]
Sep 09 22:00:00.171764 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.171295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq"
Sep 09 22:00:00.172261 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.172239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"]
Sep 09 22:00:00.177294 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.177106    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"
Sep 09 22:00:00.195511 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.195479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq"]
Sep 09 22:00:00.244936 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.244899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq"
Sep 09 22:00:00.519333 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.519216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"]
Sep 09 22:00:00.521690 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.521509    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"]
Sep 09 22:00:00.593461 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.593396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq"]
Sep 09 22:00:00.595124 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:00:00.595094    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode530b8fb_835e_4edb_976f_87debade942f.slice/crio-4d81a7437440d576c4168de47905b489c307132515792978ac8dc74b2d649416 WatchSource:0}: Error finding container 4d81a7437440d576c4168de47905b489c307132515792978ac8dc74b2d649416: Status 404 returned error can't find the container with id 4d81a7437440d576c4168de47905b489c307132515792978ac8dc74b2d649416
Sep 09 22:00:00.635747 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.634528    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"]
Sep 09 22:00:00.635747 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.635101    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj" podUID="9c1edf93-16ad-4518-b3ef-84a06769fe5f" containerName="payload-pre-processing" containerID="cri-o://4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25" gracePeriod=30
Sep 09 22:00:00.797326 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.797109    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"]
Sep 09 22:00:00.799977 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.799948    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df8f95ff-mm58c"]
Sep 09 22:00:00.932917 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.932813    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 22:00:00.980518 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.980063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8j5s\" (UniqueName: \"kubernetes.io/projected/9c1edf93-16ad-4518-b3ef-84a06769fe5f-kube-api-access-k8j5s\") pod \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\" (UID: \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\") "
Sep 09 22:00:00.980518 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.980098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1edf93-16ad-4518-b3ef-84a06769fe5f-plugins-config-volume\") pod \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\" (UID: \"9c1edf93-16ad-4518-b3ef-84a06769fe5f\") "
Sep 09 22:00:00.980915 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.980888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1edf93-16ad-4518-b3ef-84a06769fe5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c1edf93-16ad-4518-b3ef-84a06769fe5f" (UID: "9c1edf93-16ad-4518-b3ef-84a06769fe5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:00:00.982905 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:00.982858    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1edf93-16ad-4518-b3ef-84a06769fe5f-kube-api-access-k8j5s" (OuterVolumeSpecName: "kube-api-access-k8j5s") pod "9c1edf93-16ad-4518-b3ef-84a06769fe5f" (UID: "9c1edf93-16ad-4518-b3ef-84a06769fe5f"). InnerVolumeSpecName "kube-api-access-k8j5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:01.081698 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.081629    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8j5s\" (UniqueName: \"kubernetes.io/projected/9c1edf93-16ad-4518-b3ef-84a06769fe5f-kube-api-access-k8j5s\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:01.081698 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.081656    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1edf93-16ad-4518-b3ef-84a06769fe5f-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:01.260206 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.260170    2579 generic.go:356] "Generic (PLEG): container finished" podID="9c1edf93-16ad-4518-b3ef-84a06769fe5f" containerID="4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25" exitCode=0
Sep 09 22:00:01.260661 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.260213    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"
Sep 09 22:00:01.260661 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.260246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj" event={"ID":"9c1edf93-16ad-4518-b3ef-84a06769fe5f","Type":"ContainerDied","Data":"4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25"}
Sep 09 22:00:01.260661 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.260304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj" event={"ID":"9c1edf93-16ad-4518-b3ef-84a06769fe5f","Type":"ContainerDied","Data":"a31f3d305edb9b8acede91807e0689534628fe34c41261b51c602dbdff5281a5"}
Sep 09 22:00:01.260661 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.260338    2579 scope.go:117] "RemoveContainer" containerID="4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25"
Sep 09 22:00:01.262043 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.262015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr" event={"ID":"4e218407-dfa7-41bb-918f-e1e56f787687","Type":"ContainerStarted","Data":"cff57fb3c6aedb33390a6466b40fc977593cbc4ea628c70a8f59fde07be066ab"}
Sep 09 22:00:01.262192 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.262067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr" event={"ID":"4e218407-dfa7-41bb-918f-e1e56f787687","Type":"ContainerStarted","Data":"3b243c4d739b9753e5a878ff4a89abf49b98c27d8a67f59922c3be86fd5da500"}
Sep 09 22:00:01.263891 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.263856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5" event={"ID":"a20807a8-ea12-4368-a3f7-571ba484b516","Type":"ContainerStarted","Data":"1b5c9c93320e3ec0ecce273d48027dbdfcf4f528448515ed89d9987612c43979"}
Sep 09 22:00:01.264018 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.263902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5" event={"ID":"a20807a8-ea12-4368-a3f7-571ba484b516","Type":"ContainerStarted","Data":"513c99476a259cd23c38c9755e2d79b51869acd66b353725de6e0ace037e2d8e"}
Sep 09 22:00:01.265591 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.265527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq" event={"ID":"e530b8fb-835e-4edb-976f-87debade942f","Type":"ContainerStarted","Data":"ff6cac2aa37e45b8c4f7093478600318e336a49b27b77ad9e04a42d37254833b"}
Sep 09 22:00:01.265591 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.265575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq" event={"ID":"e530b8fb-835e-4edb-976f-87debade942f","Type":"ContainerStarted","Data":"4d81a7437440d576c4168de47905b489c307132515792978ac8dc74b2d649416"}
Sep 09 22:00:01.274770 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.274755    2579 scope.go:117] "RemoveContainer" containerID="4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25"
Sep 09 22:00:01.275056 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:00:01.275034    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25\": container with ID starting with 4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25 not found: ID does not exist" containerID="4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25"
Sep 09 22:00:01.275120 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.275068    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25"} err="failed to get container status \"4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25\": rpc error: code = NotFound desc = could not find container \"4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25\": container with ID starting with 4b8d4c9c9f3110589e90a33bbc0d1c5977d59b50043fa5e728a29b8236aacd25 not found: ID does not exist"
Sep 09 22:00:01.282374 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.282331    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr" podStartSLOduration=1.282318625 podStartE2EDuration="1.282318625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:01.281235932 +0000 UTC m=+1275.703598632" watchObservedRunningTime="2026-09-09 22:00:01.282318625 +0000 UTC m=+1275.704681371"
Sep 09 22:00:01.310675 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.300453    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5" podStartSLOduration=1.300433123 podStartE2EDuration="1.300433123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:01.296835058 +0000 UTC m=+1275.719197758" watchObservedRunningTime="2026-09-09 22:00:01.300433123 +0000 UTC m=+1275.722795824"
Sep 09 22:00:01.324548 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.324455    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq" podStartSLOduration=1.324431715 podStartE2EDuration="1.324431715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:01.321010892 +0000 UTC m=+1275.743373594" watchObservedRunningTime="2026-09-09 22:00:01.324431715 +0000 UTC m=+1275.746794415"
Sep 09 22:00:01.338839 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.338752    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"]
Sep 09 22:00:01.343666 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:01.343639    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df8f95ff-5457b8b658-6bszj"]
Sep 09 22:00:02.093838 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:02.093797    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c1edf93-16ad-4518-b3ef-84a06769fe5f" path="/var/lib/kubelet/pods/9c1edf93-16ad-4518-b3ef-84a06769fe5f/volumes"
Sep 09 22:00:02.094147 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:02.094133    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0543f76-c9cd-4c05-8a3a-9b472a756381" path="/var/lib/kubelet/pods/e0543f76-c9cd-4c05-8a3a-9b472a756381/volumes"
Sep 09 22:00:02.274553 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:02.274461    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e218407-dfa7-41bb-918f-e1e56f787687" containerID="cff57fb3c6aedb33390a6466b40fc977593cbc4ea628c70a8f59fde07be066ab" exitCode=0
Sep 09 22:00:02.275027 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:02.274554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr" event={"ID":"4e218407-dfa7-41bb-918f-e1e56f787687","Type":"ContainerDied","Data":"cff57fb3c6aedb33390a6466b40fc977593cbc4ea628c70a8f59fde07be066ab"}
Sep 09 22:00:02.276223 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:02.276199    2579 generic.go:356] "Generic (PLEG): container finished" podID="a20807a8-ea12-4368-a3f7-571ba484b516" containerID="1b5c9c93320e3ec0ecce273d48027dbdfcf4f528448515ed89d9987612c43979" exitCode=0
Sep 09 22:00:02.276310 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:02.276267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5" event={"ID":"a20807a8-ea12-4368-a3f7-571ba484b516","Type":"ContainerDied","Data":"1b5c9c93320e3ec0ecce273d48027dbdfcf4f528448515ed89d9987612c43979"}
Sep 09 22:00:02.277880 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:02.277859    2579 generic.go:356] "Generic (PLEG): container finished" podID="e530b8fb-835e-4edb-976f-87debade942f" containerID="ff6cac2aa37e45b8c4f7093478600318e336a49b27b77ad9e04a42d37254833b" exitCode=0
Sep 09 22:00:02.277977 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:02.277890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq" event={"ID":"e530b8fb-835e-4edb-976f-87debade942f","Type":"ContainerDied","Data":"ff6cac2aa37e45b8c4f7093478600318e336a49b27b77ad9e04a42d37254833b"}
Sep 09 22:00:03.199814 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:03.199783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 22:00:03.464321 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:03.464280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq"
Sep 09 22:00:03.496528 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:03.496501    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"
Sep 09 22:00:03.556707 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:03.556674    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"
Sep 09 22:00:04.294063 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.293974    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"
Sep 09 22:00:04.294063 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.294004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr" event={"ID":"4e218407-dfa7-41bb-918f-e1e56f787687","Type":"ContainerDied","Data":"3b243c4d739b9753e5a878ff4a89abf49b98c27d8a67f59922c3be86fd5da500"}
Sep 09 22:00:04.294063 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.294059    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b243c4d739b9753e5a878ff4a89abf49b98c27d8a67f59922c3be86fd5da500"
Sep 09 22:00:04.295457 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.295438    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"
Sep 09 22:00:04.295587 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.295466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5" event={"ID":"a20807a8-ea12-4368-a3f7-571ba484b516","Type":"ContainerDied","Data":"513c99476a259cd23c38c9755e2d79b51869acd66b353725de6e0ace037e2d8e"}
Sep 09 22:00:04.295587 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.295506    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="513c99476a259cd23c38c9755e2d79b51869acd66b353725de6e0ace037e2d8e"
Sep 09 22:00:04.297223 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.297206    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq"
Sep 09 22:00:04.297223 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.297214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816520-lbsmq" event={"ID":"e530b8fb-835e-4edb-976f-87debade942f","Type":"ContainerDied","Data":"4d81a7437440d576c4168de47905b489c307132515792978ac8dc74b2d649416"}
Sep 09 22:00:04.297396 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:04.297265    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d81a7437440d576c4168de47905b489c307132515792978ac8dc74b2d649416"
Sep 09 22:00:07.319445 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:07.319413    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerID="7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d" exitCode=0
Sep 09 22:00:07.320031 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:07.319497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" event={"ID":"2d29cb97-241e-492f-92fb-d7702847c2f6","Type":"ContainerDied","Data":"7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d"}
Sep 09 22:00:07.320147 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:07.320127    2579 scope.go:117] "RemoveContainer" containerID="7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d"
Sep 09 22:00:07.321367 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:07.321335    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:00:07.321450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:07.321408    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:00:07.321450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:07.321444    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:00:07.488744 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:07.488650    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"]
Sep 09 22:00:07.488917 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:07.488895    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy" containerID="cri-o://c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea" gracePeriod=30
Sep 09 22:00:08.331655 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:08.331596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" event={"ID":"2d29cb97-241e-492f-92fb-d7702847c2f6","Type":"ContainerStarted","Data":"6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9"}
Sep 09 22:00:08.890787 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:08.890750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 22:00:08.891025 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:08.891006    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 22:00:08.891075 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:08.891053    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 22:00:09.879013 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:09.878975    2579 patch_prober.go:28] interesting pod/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 22:00:09.879396 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:09.879036    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 22:00:09.890851 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:09.890815    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 22:00:09.891022 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:09.890870    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 22:00:10.891471 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:10.891430    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 22:00:10.891961 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:10.891490    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 22:00:11.890667 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:11.890632    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 22:00:11.890861 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:11.890689    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 22:00:12.782650 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.782625    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 22:00:12.891044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.891001    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 09 22:00:12.891205 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.891067    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 09 22:00:12.927175 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-podinfo\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927175 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-data\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927219    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-envoy\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-socket\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927267    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkfnd\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-kube-api-access-dkfnd\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b4beaf-7735-4794-9c0d-298ca224bf4a-istiod-ca-cert\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-certs\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927355    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-credential-socket\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927450 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927382    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-token\") pod \"73b4beaf-7735-4794-9c0d-298ca224bf4a\" (UID: \"73b4beaf-7735-4794-9c0d-298ca224bf4a\") "
Sep 09 22:00:12.927836 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:12.927836 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927772    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:12.927836 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927788    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b4beaf-7735-4794-9c0d-298ca224bf4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:00:12.928007 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:12.928007 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.927981    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:12.928115 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.928017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:12.929704 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.929676    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 22:00:12.929840 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.929729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:12.929840 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.929745    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-kube-api-access-dkfnd" (OuterVolumeSpecName: "kube-api-access-dkfnd") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "kube-api-access-dkfnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:12.929973 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:12.929930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "73b4beaf-7735-4794-9c0d-298ca224bf4a" (UID: "73b4beaf-7735-4794-9c0d-298ca224bf4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:00:13.029044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.028956    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:13.029044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.028986    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:13.029044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.028995    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:13.029044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.029003    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:13.029044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.029011    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:13.029044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.029021    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfnd\" (UniqueName: \"kubernetes.io/projected/73b4beaf-7735-4794-9c0d-298ca224bf4a-kube-api-access-dkfnd\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:13.029044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.029029    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b4beaf-7735-4794-9c0d-298ca224bf4a-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:13.029044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.029038    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b4beaf-7735-4794-9c0d-298ca224bf4a-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:00:13.364197 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.364162    2579 generic.go:356] "Generic (PLEG): container finished" podID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerID="c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea" exitCode=0
Sep 09 22:00:13.364415 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.364208    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"
Sep 09 22:00:13.364415 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.364243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" event={"ID":"73b4beaf-7735-4794-9c0d-298ca224bf4a","Type":"ContainerDied","Data":"c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea"}
Sep 09 22:00:13.364415 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.364304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g" event={"ID":"73b4beaf-7735-4794-9c0d-298ca224bf4a","Type":"ContainerDied","Data":"b782ffa82ba621b21606e958bbf5c1c1c08933fc16b444aaa0d326115e34f8cf"}
Sep 09 22:00:13.364415 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.364326    2579 scope.go:117] "RemoveContainer" containerID="c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea"
Sep 09 22:00:13.376298 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.376272    2579 scope.go:117] "RemoveContainer" containerID="1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93"
Sep 09 22:00:13.387353 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.387254    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"]
Sep 09 22:00:13.387452 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.387429    2579 scope.go:117] "RemoveContainer" containerID="c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea"
Sep 09 22:00:13.387931 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:00:13.387866    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea\": container with ID starting with c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea not found: ID does not exist" containerID="c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea"
Sep 09 22:00:13.387931 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.387906    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea"} err="failed to get container status \"c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea\": rpc error: code = NotFound desc = could not find container \"c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea\": container with ID starting with c730d6ca42f40f0d00c8e65639642346d3ad7d7b30d4f65481b427c6082475ea not found: ID does not exist"
Sep 09 22:00:13.387931 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.387932    2579 scope.go:117] "RemoveContainer" containerID="1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93"
Sep 09 22:00:13.388407 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:00:13.388379    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93\": container with ID starting with 1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93 not found: ID does not exist" containerID="1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93"
Sep 09 22:00:13.388517 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.388416    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93"} err="failed to get container status \"1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93\": rpc error: code = NotFound desc = could not find container \"1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93\": container with ID starting with 1fec6e3a07ed732d5199861d6ed1499395493045d7426b642a57ecbb4894ee93 not found: ID does not exist"
Sep 09 22:00:13.391304 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.391265    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df8f95ff-openshift-default-54db679c48-vcg6g"]
Sep 09 22:00:13.895286 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.895255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 22:00:13.895769 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.895642    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 22:00:13.896354 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:13.896336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 22:00:14.092618 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:14.092577    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" path="/var/lib/kubelet/pods/73b4beaf-7735-4794-9c0d-298ca224bf4a/volumes"
Sep 09 22:00:26.200011 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.199968    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"]
Sep 09 22:00:26.200618 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200587    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy"
Sep 09 22:00:26.200673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200620    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy"
Sep 09 22:00:26.200673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200635    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy"
Sep 09 22:00:26.200673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200644    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy"
Sep 09 22:00:26.200673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200656    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20807a8-ea12-4368-a3f7-571ba484b516" containerName="cleanup"
Sep 09 22:00:26.200673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200665    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20807a8-ea12-4368-a3f7-571ba484b516" containerName="cleanup"
Sep 09 22:00:26.200832 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200676    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e218407-dfa7-41bb-918f-e1e56f787687" containerName="cleanup"
Sep 09 22:00:26.200832 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200684    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e218407-dfa7-41bb-918f-e1e56f787687" containerName="cleanup"
Sep 09 22:00:26.200832 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200698    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e530b8fb-835e-4edb-976f-87debade942f" containerName="cleanup"
Sep 09 22:00:26.200832 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200706    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e530b8fb-835e-4edb-976f-87debade942f" containerName="cleanup"
Sep 09 22:00:26.200832 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200738    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1edf93-16ad-4518-b3ef-84a06769fe5f" containerName="payload-pre-processing"
Sep 09 22:00:26.200832 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200748    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1edf93-16ad-4518-b3ef-84a06769fe5f" containerName="payload-pre-processing"
Sep 09 22:00:26.201016 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200834    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e530b8fb-835e-4edb-976f-87debade942f" containerName="cleanup"
Sep 09 22:00:26.201016 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200848    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e218407-dfa7-41bb-918f-e1e56f787687" containerName="cleanup"
Sep 09 22:00:26.201016 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200858    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c1edf93-16ad-4518-b3ef-84a06769fe5f" containerName="payload-pre-processing"
Sep 09 22:00:26.201016 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200869    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy"
Sep 09 22:00:26.201016 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.200877    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20807a8-ea12-4368-a3f7-571ba484b516" containerName="cleanup"
Sep 09 22:00:26.204008 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.203989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:26.207073 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.207051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f6cce2-dockercfg-sh2jt\""
Sep 09 22:00:26.207260 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.207240    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f6cce2\""
Sep 09 22:00:26.212890 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.212853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"]
Sep 09 22:00:26.243463 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.243426    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"]
Sep 09 22:00:26.244027 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.244008    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b4beaf-7735-4794-9c0d-298ca224bf4a" containerName="istio-proxy"
Sep 09 22:00:26.246018 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.245997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:26.255809 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.255785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"]
Sep 09 22:00:26.264835 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.264813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh\" (UID: \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:26.264942 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.264882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qbj\" (UniqueName: \"kubernetes.io/projected/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-kube-api-access-d5qbj\") pod \"payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh\" (UID: \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:26.365688 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.365659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01acd97a-bcae-4005-be37-764a83892734-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df\" (UID: \"01acd97a-bcae-4005-be37-764a83892734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:26.365874 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.365705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qbj\" (UniqueName: \"kubernetes.io/projected/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-kube-api-access-d5qbj\") pod \"payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh\" (UID: \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:26.365874 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.365835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxk8\" (UniqueName: \"kubernetes.io/projected/01acd97a-bcae-4005-be37-764a83892734-kube-api-access-gcxk8\") pod \"payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df\" (UID: \"01acd97a-bcae-4005-be37-764a83892734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:26.365955 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.365918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh\" (UID: \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:26.366586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.366568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh\" (UID: \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:26.376193 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.376172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qbj\" (UniqueName: \"kubernetes.io/projected/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-kube-api-access-d5qbj\") pod \"payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh\" (UID: \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:26.466862 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.466759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxk8\" (UniqueName: \"kubernetes.io/projected/01acd97a-bcae-4005-be37-764a83892734-kube-api-access-gcxk8\") pod \"payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df\" (UID: \"01acd97a-bcae-4005-be37-764a83892734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:26.467044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.466875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01acd97a-bcae-4005-be37-764a83892734-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df\" (UID: \"01acd97a-bcae-4005-be37-764a83892734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:26.467568 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.467546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01acd97a-bcae-4005-be37-764a83892734-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df\" (UID: \"01acd97a-bcae-4005-be37-764a83892734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:26.475560 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.475536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxk8\" (UniqueName: \"kubernetes.io/projected/01acd97a-bcae-4005-be37-764a83892734-kube-api-access-gcxk8\") pod \"payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df\" (UID: \"01acd97a-bcae-4005-be37-764a83892734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:26.515614 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.515575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:26.557756 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.557722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:26.789815 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.789790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"]
Sep 09 22:00:26.792700 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:00:26.792669    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e6ae76_f05a_4af3_be25_8c32d0d45a7c.slice/crio-a290016284635b5d429bab0ed14f71f54fa30208ed2e57952907660a1a69fa9b WatchSource:0}: Error finding container a290016284635b5d429bab0ed14f71f54fa30208ed2e57952907660a1a69fa9b: Status 404 returned error can't find the container with id a290016284635b5d429bab0ed14f71f54fa30208ed2e57952907660a1a69fa9b
Sep 09 22:00:26.848298 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:26.848275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"]
Sep 09 22:00:26.849415 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:00:26.849384    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01acd97a_bcae_4005_be37_764a83892734.slice/crio-bccde30f50e4f0e72210e46a6d2c26bccc5db4f06c770fb42c3a36a3e635d6be WatchSource:0}: Error finding container bccde30f50e4f0e72210e46a6d2c26bccc5db4f06c770fb42c3a36a3e635d6be: Status 404 returned error can't find the container with id bccde30f50e4f0e72210e46a6d2c26bccc5db4f06c770fb42c3a36a3e635d6be
Sep 09 22:00:27.454393 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:27.454353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh" event={"ID":"22e6ae76-f05a-4af3-be25-8c32d0d45a7c","Type":"ContainerStarted","Data":"21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef"}
Sep 09 22:00:27.454895 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:27.454430    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:27.454895 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:27.454453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh" event={"ID":"22e6ae76-f05a-4af3-be25-8c32d0d45a7c","Type":"ContainerStarted","Data":"a290016284635b5d429bab0ed14f71f54fa30208ed2e57952907660a1a69fa9b"}
Sep 09 22:00:27.456001 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:27.455973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df" event={"ID":"01acd97a-bcae-4005-be37-764a83892734","Type":"ContainerStarted","Data":"a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c"}
Sep 09 22:00:27.456124 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:27.456023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df" event={"ID":"01acd97a-bcae-4005-be37-764a83892734","Type":"ContainerStarted","Data":"bccde30f50e4f0e72210e46a6d2c26bccc5db4f06c770fb42c3a36a3e635d6be"}
Sep 09 22:00:27.456124 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:27.456053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:27.471618 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:27.471562    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh" podStartSLOduration=1.471546147 podStartE2EDuration="1.471546147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:27.470241772 +0000 UTC m=+1301.892604505" watchObservedRunningTime="2026-09-09 22:00:27.471546147 +0000 UTC m=+1301.893908847"
Sep 09 22:00:27.486786 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:27.486736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df" podStartSLOduration=1.486719152 podStartE2EDuration="1.486719152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:27.486155473 +0000 UTC m=+1301.908518174" watchObservedRunningTime="2026-09-09 22:00:27.486719152 +0000 UTC m=+1301.909081889"
Sep 09 22:00:38.465088 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:38.465057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:00:38.465518 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:38.465152    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:00:55.357522 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.357487    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"]
Sep 09 22:00:55.361255 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.361232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:55.363924 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.363898    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e5cb43-dockercfg-9zdmp\""
Sep 09 22:00:55.363924 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.363902    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e5cb43\""
Sep 09 22:00:55.371739 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.371713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"]
Sep 09 22:00:55.455857 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.455823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c538dd-d704-4390-a26f-bdefe2203253-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms\" (UID: \"07c538dd-d704-4390-a26f-bdefe2203253\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:55.456042 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.455903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54dvz\" (UniqueName: \"kubernetes.io/projected/07c538dd-d704-4390-a26f-bdefe2203253-kube-api-access-54dvz\") pod \"payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms\" (UID: \"07c538dd-d704-4390-a26f-bdefe2203253\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:55.557104 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.557061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c538dd-d704-4390-a26f-bdefe2203253-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms\" (UID: \"07c538dd-d704-4390-a26f-bdefe2203253\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:55.557295 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.557122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54dvz\" (UniqueName: \"kubernetes.io/projected/07c538dd-d704-4390-a26f-bdefe2203253-kube-api-access-54dvz\") pod \"payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms\" (UID: \"07c538dd-d704-4390-a26f-bdefe2203253\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:55.557842 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.557815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c538dd-d704-4390-a26f-bdefe2203253-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms\" (UID: \"07c538dd-d704-4390-a26f-bdefe2203253\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:55.565720 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.565695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54dvz\" (UniqueName: \"kubernetes.io/projected/07c538dd-d704-4390-a26f-bdefe2203253-kube-api-access-54dvz\") pod \"payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms\" (UID: \"07c538dd-d704-4390-a26f-bdefe2203253\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:55.672253 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.672151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:55.944452 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:55.944378    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"]
Sep 09 22:00:55.947798 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:00:55.947764    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c538dd_d704_4390_a26f_bdefe2203253.slice/crio-c3fe0181af8702605d1829820c835a07e458348af8e46ff8e91c8358b7564eda WatchSource:0}: Error finding container c3fe0181af8702605d1829820c835a07e458348af8e46ff8e91c8358b7564eda: Status 404 returned error can't find the container with id c3fe0181af8702605d1829820c835a07e458348af8e46ff8e91c8358b7564eda
Sep 09 22:00:56.638978 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:56.638937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms" event={"ID":"07c538dd-d704-4390-a26f-bdefe2203253","Type":"ContainerStarted","Data":"8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad"}
Sep 09 22:00:56.639380 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:56.639027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms" event={"ID":"07c538dd-d704-4390-a26f-bdefe2203253","Type":"ContainerStarted","Data":"c3fe0181af8702605d1829820c835a07e458348af8e46ff8e91c8358b7564eda"}
Sep 09 22:00:56.639380 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:56.639069    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:00:56.664256 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:00:56.664195    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms" podStartSLOduration=1.664179072 podStartE2EDuration="1.664179072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:56.661672699 +0000 UTC m=+1331.084035393" watchObservedRunningTime="2026-09-09 22:00:56.664179072 +0000 UTC m=+1331.086541772"
Sep 09 22:01:07.646478 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:07.646447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:01:31.712727 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.712689    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"]
Sep 09 22:01:31.716805 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.716786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.719709 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.719688    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6a897d5c-openshift-default-dockercfg-6zmcr\""
Sep 09 22:01:31.728659 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.728633    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"]
Sep 09 22:01:31.830330 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5039b1-bf1d-4743-808c-203161aee555-istiod-ca-cert\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.830532 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7l72\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-kube-api-access-h7l72\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.830532 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-socket\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.830532 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-certs\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.830761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5039b1-bf1d-4743-808c-203161aee555-istio-podinfo\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.830761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-credential-socket\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.830761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-envoy\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.830761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-istio-token\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.830761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.830743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-data\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.931486 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-socket\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.931724 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-certs\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.931724 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5039b1-bf1d-4743-808c-203161aee555-istio-podinfo\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.931724 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-credential-socket\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.931724 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-envoy\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.931724 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-istio-token\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.932004 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-data\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.932004 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5039b1-bf1d-4743-808c-203161aee555-istiod-ca-cert\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.932004 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7l72\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-kube-api-access-h7l72\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.932004 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.931933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-socket\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.932250 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.932230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-certs\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.932338 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.932310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-credential-socket\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.932338 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.932331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-data\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.932853 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.932823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5039b1-bf1d-4743-808c-203161aee555-istiod-ca-cert\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.934149 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.934125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-envoy\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.934237 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.934220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5039b1-bf1d-4743-808c-203161aee555-istio-podinfo\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.939550 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.939527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-istio-token\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:31.940349 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:31.940331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7l72\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-kube-api-access-h7l72\") pod \"e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") " pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:32.028516 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:32.028430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:32.306425 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:32.306288    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"]
Sep 09 22:01:32.309463 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:01:32.309434    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5039b1_bf1d_4743_808c_203161aee555.slice/crio-b951f375f2feaf2a153dbfd87f9f3eef1555089cdfb1557edc341db857d6ce7f WatchSource:0}: Error finding container b951f375f2feaf2a153dbfd87f9f3eef1555089cdfb1557edc341db857d6ce7f: Status 404 returned error can't find the container with id b951f375f2feaf2a153dbfd87f9f3eef1555089cdfb1557edc341db857d6ce7f
Sep 09 22:01:32.311805 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:32.311770    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:01:32.311910 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:32.311833    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:01:32.311910 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:32.311880    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:01:32.856244 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:32.856204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" event={"ID":"bf5039b1-bf1d-4743-808c-203161aee555","Type":"ContainerStarted","Data":"5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d"}
Sep 09 22:01:32.856635 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:32.856274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" event={"ID":"bf5039b1-bf1d-4743-808c-203161aee555","Type":"ContainerStarted","Data":"b951f375f2feaf2a153dbfd87f9f3eef1555089cdfb1557edc341db857d6ce7f"}
Sep 09 22:01:32.879673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:32.879629    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podStartSLOduration=1.879596717 podStartE2EDuration="1.879596717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:01:32.877890983 +0000 UTC m=+1367.300253683" watchObservedRunningTime="2026-09-09 22:01:32.879596717 +0000 UTC m=+1367.301959418"
Sep 09 22:01:33.028565 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:33.028524    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:33.034251 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:33.034226    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:33.862095 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:33.862061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:33.863171 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:33.863151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:36.825374 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.825339    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"]
Sep 09 22:01:36.829180 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.829162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:36.831614 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.831579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6a897d5c\""
Sep 09 22:01:36.840631 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.840581    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"]
Sep 09 22:01:36.856909 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.856876    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"]
Sep 09 22:01:36.860664 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.860640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:36.863194 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.863176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6a897d5c-dockercfg-546pr\""
Sep 09 22:01:36.867585 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.867548    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"]
Sep 09 22:01:36.884046 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.884010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0be3390-211e-4154-a690-b489ee4425c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d\" (UID: \"a0be3390-211e-4154-a690-b489ee4425c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:36.884208 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.884068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ll9\" (UniqueName: \"kubernetes.io/projected/a0be3390-211e-4154-a690-b489ee4425c2-kube-api-access-r5ll9\") pod \"payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d\" (UID: \"a0be3390-211e-4154-a690-b489ee4425c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:36.985237 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.985194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0be3390-211e-4154-a690-b489ee4425c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d\" (UID: \"a0be3390-211e-4154-a690-b489ee4425c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:36.985461 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.985260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zqb\" (UniqueName: \"kubernetes.io/projected/45ece9ec-7182-445e-bb2d-f2ee84df3577-kube-api-access-t5zqb\") pod \"payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h\" (UID: \"45ece9ec-7182-445e-bb2d-f2ee84df3577\") " pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:36.985461 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.985321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ll9\" (UniqueName: \"kubernetes.io/projected/a0be3390-211e-4154-a690-b489ee4425c2-kube-api-access-r5ll9\") pod \"payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d\" (UID: \"a0be3390-211e-4154-a690-b489ee4425c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:36.985461 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.985417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ece9ec-7182-445e-bb2d-f2ee84df3577-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h\" (UID: \"45ece9ec-7182-445e-bb2d-f2ee84df3577\") " pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:36.986538 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.986511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0be3390-211e-4154-a690-b489ee4425c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d\" (UID: \"a0be3390-211e-4154-a690-b489ee4425c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:36.994982 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:36.994954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ll9\" (UniqueName: \"kubernetes.io/projected/a0be3390-211e-4154-a690-b489ee4425c2-kube-api-access-r5ll9\") pod \"payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d\" (UID: \"a0be3390-211e-4154-a690-b489ee4425c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:37.086685 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.086578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zqb\" (UniqueName: \"kubernetes.io/projected/45ece9ec-7182-445e-bb2d-f2ee84df3577-kube-api-access-t5zqb\") pod \"payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h\" (UID: \"45ece9ec-7182-445e-bb2d-f2ee84df3577\") " pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:37.086685 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.086679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ece9ec-7182-445e-bb2d-f2ee84df3577-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h\" (UID: \"45ece9ec-7182-445e-bb2d-f2ee84df3577\") " pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:37.087507 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.087487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ece9ec-7182-445e-bb2d-f2ee84df3577-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h\" (UID: \"45ece9ec-7182-445e-bb2d-f2ee84df3577\") " pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:37.094634 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.094593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zqb\" (UniqueName: \"kubernetes.io/projected/45ece9ec-7182-445e-bb2d-f2ee84df3577-kube-api-access-t5zqb\") pod \"payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h\" (UID: \"45ece9ec-7182-445e-bb2d-f2ee84df3577\") " pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:37.140337 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.140297    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:37.173783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.173748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:37.414096 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.414072    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"]
Sep 09 22:01:37.415847 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:01:37.415819    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0be3390_211e_4154_a690_b489ee4425c2.slice/crio-51beb52f38ccc33eb1fff42a163e6513bb90b1dc446ff90cc16e183902c5c5c5 WatchSource:0}: Error finding container 51beb52f38ccc33eb1fff42a163e6513bb90b1dc446ff90cc16e183902c5c5c5: Status 404 returned error can't find the container with id 51beb52f38ccc33eb1fff42a163e6513bb90b1dc446ff90cc16e183902c5c5c5
Sep 09 22:01:37.459933 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.459902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"]
Sep 09 22:01:37.460973 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:01:37.460947    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ece9ec_7182_445e_bb2d_f2ee84df3577.slice/crio-f59a4e2b85df3a67e937524eab37b3ea16e078a58be805c33b35229417c890e7 WatchSource:0}: Error finding container f59a4e2b85df3a67e937524eab37b3ea16e078a58be805c33b35229417c890e7: Status 404 returned error can't find the container with id f59a4e2b85df3a67e937524eab37b3ea16e078a58be805c33b35229417c890e7
Sep 09 22:01:37.887966 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.887924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h" event={"ID":"45ece9ec-7182-445e-bb2d-f2ee84df3577","Type":"ContainerStarted","Data":"529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb"}
Sep 09 22:01:37.888431 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.888127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:37.888431 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.888163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h" event={"ID":"45ece9ec-7182-445e-bb2d-f2ee84df3577","Type":"ContainerStarted","Data":"f59a4e2b85df3a67e937524eab37b3ea16e078a58be805c33b35229417c890e7"}
Sep 09 22:01:37.889643 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.889591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d" event={"ID":"a0be3390-211e-4154-a690-b489ee4425c2","Type":"ContainerStarted","Data":"949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277"}
Sep 09 22:01:37.889775 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.889657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d" event={"ID":"a0be3390-211e-4154-a690-b489ee4425c2","Type":"ContainerStarted","Data":"51beb52f38ccc33eb1fff42a163e6513bb90b1dc446ff90cc16e183902c5c5c5"}
Sep 09 22:01:37.889775 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.889677    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:37.906060 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.906004    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h" podStartSLOduration=1.905983783 podStartE2EDuration="1.905983783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:01:37.903501899 +0000 UTC m=+1372.325864601" watchObservedRunningTime="2026-09-09 22:01:37.905983783 +0000 UTC m=+1372.328346546"
Sep 09 22:01:37.920596 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:37.920541    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d" podStartSLOduration=1.920522406 podStartE2EDuration="1.920522406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:01:37.920118286 +0000 UTC m=+1372.342480987" watchObservedRunningTime="2026-09-09 22:01:37.920522406 +0000 UTC m=+1372.342885105"
Sep 09 22:01:46.236673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:46.236645    2579 scope.go:117] "RemoveContainer" containerID="06ff7a4eb846d6ce0f3cadbcedc8e1718807a0934b17c778b1da7ca949569fee"
Sep 09 22:01:48.897622 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:48.897575    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:01:48.898029 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:48.897763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:01:58.012045 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:58.012010    2579 generic.go:356] "Generic (PLEG): container finished" podID="bf5039b1-bf1d-4743-808c-203161aee555" containerID="5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d" exitCode=0
Sep 09 22:01:58.012558 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:58.012080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" event={"ID":"bf5039b1-bf1d-4743-808c-203161aee555","Type":"ContainerDied","Data":"5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d"}
Sep 09 22:01:58.012558 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:58.012502    2579 scope.go:117] "RemoveContainer" containerID="5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d"
Sep 09 22:01:58.013583 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:58.013550    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:01:58.013691 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:58.013657    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:01:58.013745 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:58.013714    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:01:59.019693 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:59.019657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" event={"ID":"bf5039b1-bf1d-4743-808c-203161aee555","Type":"ContainerStarted","Data":"6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c"}
Sep 09 22:01:59.029599 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:59.029576    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:01:59.029801 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:59.029782    2579 patch_prober.go:28] interesting pod/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 22:01:59.029848 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:01:59.029823    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 22:02:00.029060 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:00.029026    2579 patch_prober.go:28] interesting pod/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 22:02:00.029500 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:00.029077    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 22:02:01.028785 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:01.028739    2579 patch_prober.go:28] interesting pod/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 22:02:01.029018 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:01.028803    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 22:02:02.029190 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:02.029153    2579 patch_prober.go:28] interesting pod/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 22:02:02.029633 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:02.029219    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 22:02:03.029002 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:03.028959    2579 patch_prober.go:28] interesting pod/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 22:02:03.029214 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:03.029036    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 22:02:03.862269 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:03.862232    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:02:04.029730 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:04.029684    2579 patch_prober.go:28] interesting pod/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 22:02:04.030150 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:04.029747    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 22:02:05.033110 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:05.033077    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:02:05.034302 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:05.034278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:02:38.968436 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:38.968350    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"]
Sep 09 22:02:38.971434 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:38.971417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"
Sep 09 22:02:38.974024 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:38.974002    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:02:38.980333 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:38.979686    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"]
Sep 09 22:02:39.049685 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:39.049645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5469077-ed75-445a-8a66-bad1c33d9b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9\" (UID: \"f5469077-ed75-445a-8a66-bad1c33d9b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"
Sep 09 22:02:39.150588 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:39.150552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5469077-ed75-445a-8a66-bad1c33d9b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9\" (UID: \"f5469077-ed75-445a-8a66-bad1c33d9b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"
Sep 09 22:02:39.151161 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:39.151140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5469077-ed75-445a-8a66-bad1c33d9b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9\" (UID: \"f5469077-ed75-445a-8a66-bad1c33d9b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"
Sep 09 22:02:39.284189 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:39.284104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"
Sep 09 22:02:39.552674 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:39.552648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"]
Sep 09 22:02:39.554560 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:02:39.554529    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5469077_ed75_445a_8a66_bad1c33d9b0f.slice/crio-bf53cb96c9f3ade2c66cdc0e16214a31e08cea3b718777c968bca92ba1976d68 WatchSource:0}: Error finding container bf53cb96c9f3ade2c66cdc0e16214a31e08cea3b718777c968bca92ba1976d68: Status 404 returned error can't find the container with id bf53cb96c9f3ade2c66cdc0e16214a31e08cea3b718777c968bca92ba1976d68
Sep 09 22:02:40.261580 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:40.261543    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5469077-ed75-445a-8a66-bad1c33d9b0f" containerID="40cba332ec672d2042a0f5fa85f644d9cabffc400bc7c0ace0bbc3d72af76a45" exitCode=0
Sep 09 22:02:40.262059 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:40.261591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9" event={"ID":"f5469077-ed75-445a-8a66-bad1c33d9b0f","Type":"ContainerDied","Data":"40cba332ec672d2042a0f5fa85f644d9cabffc400bc7c0ace0bbc3d72af76a45"}
Sep 09 22:02:40.262059 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:40.261663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9" event={"ID":"f5469077-ed75-445a-8a66-bad1c33d9b0f","Type":"ContainerStarted","Data":"bf53cb96c9f3ade2c66cdc0e16214a31e08cea3b718777c968bca92ba1976d68"}
Sep 09 22:02:41.407216 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:41.407186    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"
Sep 09 22:02:41.474395 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:41.474360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5469077-ed75-445a-8a66-bad1c33d9b0f-maas-api-service-ca\") pod \"f5469077-ed75-445a-8a66-bad1c33d9b0f\" (UID: \"f5469077-ed75-445a-8a66-bad1c33d9b0f\") "
Sep 09 22:02:41.474789 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:41.474764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5469077-ed75-445a-8a66-bad1c33d9b0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5469077-ed75-445a-8a66-bad1c33d9b0f" (UID: "f5469077-ed75-445a-8a66-bad1c33d9b0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:02:41.575848 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:41.575760    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5469077-ed75-445a-8a66-bad1c33d9b0f-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:02:42.275303 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:42.275273    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"
Sep 09 22:02:42.275511 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:42.275313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9" event={"ID":"f5469077-ed75-445a-8a66-bad1c33d9b0f","Type":"ContainerDied","Data":"bf53cb96c9f3ade2c66cdc0e16214a31e08cea3b718777c968bca92ba1976d68"}
Sep 09 22:02:42.275511 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:42.275370    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf53cb96c9f3ade2c66cdc0e16214a31e08cea3b718777c968bca92ba1976d68"
Sep 09 22:02:46.257979 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:46.257944    2579 scope.go:117] "RemoveContainer" containerID="e9a61f2744f9d28fb79f2f784e4f29ef35bd233a7c89227c405e9b6aee7bb225"
Sep 09 22:02:46.267293 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:46.267266    2579 scope.go:117] "RemoveContainer" containerID="8ea4a0d4183c464c66b2c9bf91d28a77ca4545784840d806b4959c6eaac26da9"
Sep 09 22:02:46.276165 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:46.276138    2579 scope.go:117] "RemoveContainer" containerID="bb26d0a7be366ba90648b1b65fdb106e8649592d5e08a773d2031468f7140f40"
Sep 09 22:02:54.520304 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.520266    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"]
Sep 09 22:02:54.520795 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.520529    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h" podUID="45ece9ec-7182-445e-bb2d-f2ee84df3577" containerName="payload-processing" containerID="cri-o://529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb" gracePeriod=30
Sep 09 22:02:54.534513 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.534485    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"]
Sep 09 22:02:54.534731 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.534708    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d" podUID="a0be3390-211e-4154-a690-b489ee4425c2" containerName="payload-pre-processing" containerID="cri-o://949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277" gracePeriod=30
Sep 09 22:02:54.787284 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.787258    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:02:54.797769 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.797738    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"]
Sep 09 22:02:54.804010 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.803985    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a897d5c-tgdd9"]
Sep 09 22:02:54.826350 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.826327    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:02:54.922648 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.922598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5zqb\" (UniqueName: \"kubernetes.io/projected/45ece9ec-7182-445e-bb2d-f2ee84df3577-kube-api-access-t5zqb\") pod \"45ece9ec-7182-445e-bb2d-f2ee84df3577\" (UID: \"45ece9ec-7182-445e-bb2d-f2ee84df3577\") "
Sep 09 22:02:54.922848 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.922674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5ll9\" (UniqueName: \"kubernetes.io/projected/a0be3390-211e-4154-a690-b489ee4425c2-kube-api-access-r5ll9\") pod \"a0be3390-211e-4154-a690-b489ee4425c2\" (UID: \"a0be3390-211e-4154-a690-b489ee4425c2\") "
Sep 09 22:02:54.922848 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.922764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ece9ec-7182-445e-bb2d-f2ee84df3577-plugins-config-volume\") pod \"45ece9ec-7182-445e-bb2d-f2ee84df3577\" (UID: \"45ece9ec-7182-445e-bb2d-f2ee84df3577\") "
Sep 09 22:02:54.922848 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.922806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0be3390-211e-4154-a690-b489ee4425c2-plugins-config-volume\") pod \"a0be3390-211e-4154-a690-b489ee4425c2\" (UID: \"a0be3390-211e-4154-a690-b489ee4425c2\") "
Sep 09 22:02:54.923125 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.923092    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ece9ec-7182-445e-bb2d-f2ee84df3577-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45ece9ec-7182-445e-bb2d-f2ee84df3577" (UID: "45ece9ec-7182-445e-bb2d-f2ee84df3577"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:02:54.923251 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.923195    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ece9ec-7182-445e-bb2d-f2ee84df3577-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:02:54.923338 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.923302    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0be3390-211e-4154-a690-b489ee4425c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0be3390-211e-4154-a690-b489ee4425c2" (UID: "a0be3390-211e-4154-a690-b489ee4425c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:02:54.925253 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.925216    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ece9ec-7182-445e-bb2d-f2ee84df3577-kube-api-access-t5zqb" (OuterVolumeSpecName: "kube-api-access-t5zqb") pod "45ece9ec-7182-445e-bb2d-f2ee84df3577" (UID: "45ece9ec-7182-445e-bb2d-f2ee84df3577"). InnerVolumeSpecName "kube-api-access-t5zqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:54.925642 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:54.925586    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0be3390-211e-4154-a690-b489ee4425c2-kube-api-access-r5ll9" (OuterVolumeSpecName: "kube-api-access-r5ll9") pod "a0be3390-211e-4154-a690-b489ee4425c2" (UID: "a0be3390-211e-4154-a690-b489ee4425c2"). InnerVolumeSpecName "kube-api-access-r5ll9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:02:55.024182 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.024103    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zqb\" (UniqueName: \"kubernetes.io/projected/45ece9ec-7182-445e-bb2d-f2ee84df3577-kube-api-access-t5zqb\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:02:55.024182 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.024133    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5ll9\" (UniqueName: \"kubernetes.io/projected/a0be3390-211e-4154-a690-b489ee4425c2-kube-api-access-r5ll9\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:02:55.024182 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.024145    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0be3390-211e-4154-a690-b489ee4425c2-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:02:55.366560 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.366523    2579 generic.go:356] "Generic (PLEG): container finished" podID="45ece9ec-7182-445e-bb2d-f2ee84df3577" containerID="529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb" exitCode=0
Sep 09 22:02:55.366764 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.366575    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"
Sep 09 22:02:55.366764 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.366619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h" event={"ID":"45ece9ec-7182-445e-bb2d-f2ee84df3577","Type":"ContainerDied","Data":"529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb"}
Sep 09 22:02:55.366764 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.366683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h" event={"ID":"45ece9ec-7182-445e-bb2d-f2ee84df3577","Type":"ContainerDied","Data":"f59a4e2b85df3a67e937524eab37b3ea16e078a58be805c33b35229417c890e7"}
Sep 09 22:02:55.366764 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.366709    2579 scope.go:117] "RemoveContainer" containerID="529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb"
Sep 09 22:02:55.368155 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.368124    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0be3390-211e-4154-a690-b489ee4425c2" containerID="949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277" exitCode=0
Sep 09 22:02:55.368277 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.368159    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"
Sep 09 22:02:55.368277 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.368164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d" event={"ID":"a0be3390-211e-4154-a690-b489ee4425c2","Type":"ContainerDied","Data":"949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277"}
Sep 09 22:02:55.368277 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.368213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d" event={"ID":"a0be3390-211e-4154-a690-b489ee4425c2","Type":"ContainerDied","Data":"51beb52f38ccc33eb1fff42a163e6513bb90b1dc446ff90cc16e183902c5c5c5"}
Sep 09 22:02:55.378123 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.378106    2579 scope.go:117] "RemoveContainer" containerID="529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb"
Sep 09 22:02:55.378463 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:02:55.378439    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb\": container with ID starting with 529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb not found: ID does not exist" containerID="529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb"
Sep 09 22:02:55.378536 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.378471    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb"} err="failed to get container status \"529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb\": rpc error: code = NotFound desc = could not find container \"529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb\": container with ID starting with 529a23c90802edd4c87c103d879f39fc963a435144e963c37e5006aa934d4ecb not found: ID does not exist"
Sep 09 22:02:55.378536 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.378486    2579 scope.go:117] "RemoveContainer" containerID="949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277"
Sep 09 22:02:55.387052 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.387018    2579 scope.go:117] "RemoveContainer" containerID="949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277"
Sep 09 22:02:55.387371 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:02:55.387340    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277\": container with ID starting with 949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277 not found: ID does not exist" containerID="949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277"
Sep 09 22:02:55.387467 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.387380    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277"} err="failed to get container status \"949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277\": rpc error: code = NotFound desc = could not find container \"949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277\": container with ID starting with 949720f3239ee0a327e2608e8e88f5d8d952940f905acce1bbe0d1dccab69277 not found: ID does not exist"
Sep 09 22:02:55.390688 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.390669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"]
Sep 09 22:02:55.392895 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.392875    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a897d5c-68dd568fd5-k8h4h"]
Sep 09 22:02:55.403364 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.403340    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"]
Sep 09 22:02:55.406932 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:55.406909    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a897d5c-74cffb5d74-zdb6d"]
Sep 09 22:02:56.092654 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:56.092590    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ece9ec-7182-445e-bb2d-f2ee84df3577" path="/var/lib/kubelet/pods/45ece9ec-7182-445e-bb2d-f2ee84df3577/volumes"
Sep 09 22:02:56.093037 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:56.093020    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0be3390-211e-4154-a690-b489ee4425c2" path="/var/lib/kubelet/pods/a0be3390-211e-4154-a690-b489ee4425c2/volumes"
Sep 09 22:02:56.093319 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:02:56.093305    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5469077-ed75-445a-8a66-bad1c33d9b0f" path="/var/lib/kubelet/pods/f5469077-ed75-445a-8a66-bad1c33d9b0f/volumes"
Sep 09 22:03:46.321175 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:03:46.321147    2579 scope.go:117] "RemoveContainer" containerID="ca5d1d17783b2fa5c47cf1aaa4a67268bc0010bc5e4217fd5d562866f6a7294e"
Sep 09 22:04:18.379759 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:18.379675    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"]
Sep 09 22:04:18.380254 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:18.379936    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy" containerID="cri-o://6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c" gracePeriod=30
Sep 09 22:04:20.036253 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:20.036208    2579 patch_prober.go:28] interesting pod/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 22:04:20.036658 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:20.036287    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 22:04:23.642122 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.642093    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:04:23.753128 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753040    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-istio-token\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753303 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5039b1-bf1d-4743-808c-203161aee555-istio-podinfo\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753303 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753164    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-socket\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753303 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753186    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5039b1-bf1d-4743-808c-203161aee555-istiod-ca-cert\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753303 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753214    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7l72\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-kube-api-access-h7l72\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753303 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753242    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-envoy\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753303 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753267    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-credential-socket\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753689 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-certs\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753689 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-data\") pod \"bf5039b1-bf1d-4743-808c-203161aee555\" (UID: \"bf5039b1-bf1d-4743-808c-203161aee555\") "
Sep 09 22:04:23.753689 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753580    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:23.753689 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:23.754105 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5039b1-bf1d-4743-808c-203161aee555-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:04:23.754105 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753830    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:23.754105 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:23.754105 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753958    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.754105 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753984    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5039b1-bf1d-4743-808c-203161aee555-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.754105 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.753998    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.754105 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.754012    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.754105 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.754026    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.755596 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.755572    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-kube-api-access-h7l72" (OuterVolumeSpecName: "kube-api-access-h7l72") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "kube-api-access-h7l72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:04:23.755702 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.755590    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:04:23.755867 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.755845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf5039b1-bf1d-4743-808c-203161aee555-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 22:04:23.755968 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.755872    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf5039b1-bf1d-4743-808c-203161aee555" (UID: "bf5039b1-bf1d-4743-808c-203161aee555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:04:23.854926 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.854887    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.854926 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.854920    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5039b1-bf1d-4743-808c-203161aee555-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.854926 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.854930    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7l72\" (UniqueName: \"kubernetes.io/projected/bf5039b1-bf1d-4743-808c-203161aee555-kube-api-access-h7l72\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.855175 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.854940    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5039b1-bf1d-4743-808c-203161aee555-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:04:23.885352 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.885305    2579 generic.go:356] "Generic (PLEG): container finished" podID="bf5039b1-bf1d-4743-808c-203161aee555" containerID="6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c" exitCode=0
Sep 09 22:04:23.885530 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.885357    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"
Sep 09 22:04:23.885530 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.885382    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" event={"ID":"bf5039b1-bf1d-4743-808c-203161aee555","Type":"ContainerDied","Data":"6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c"}
Sep 09 22:04:23.885530 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.885456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz" event={"ID":"bf5039b1-bf1d-4743-808c-203161aee555","Type":"ContainerDied","Data":"b951f375f2feaf2a153dbfd87f9f3eef1555089cdfb1557edc341db857d6ce7f"}
Sep 09 22:04:23.885530 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.885471    2579 scope.go:117] "RemoveContainer" containerID="6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c"
Sep 09 22:04:23.895786 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.895768    2579 scope.go:117] "RemoveContainer" containerID="5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d"
Sep 09 22:04:23.906072 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.906048    2579 scope.go:117] "RemoveContainer" containerID="6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c"
Sep 09 22:04:23.906457 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:04:23.906431    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c\": container with ID starting with 6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c not found: ID does not exist" containerID="6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c"
Sep 09 22:04:23.906575 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.906466    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c"} err="failed to get container status \"6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c\": rpc error: code = NotFound desc = could not find container \"6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c\": container with ID starting with 6633e38579a1d68d468a23444bdb2e704b1e1030740eb47e3bab52fba5e5ed7c not found: ID does not exist"
Sep 09 22:04:23.906575 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.906485    2579 scope.go:117] "RemoveContainer" containerID="5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d"
Sep 09 22:04:23.906801 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:04:23.906741    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d\": container with ID starting with 5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d not found: ID does not exist" containerID="5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d"
Sep 09 22:04:23.906801 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.906763    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d"} err="failed to get container status \"5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d\": rpc error: code = NotFound desc = could not find container \"5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d\": container with ID starting with 5894620eb2aeef27d3a9cf2e6d3bf26fe1f18799677ce73b514afca58cc9907d not found: ID does not exist"
Sep 09 22:04:23.909659 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.909639    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"]
Sep 09 22:04:23.913233 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:23.913212    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6a897d5c-openshift-default-7b59bfb58d-6lpfz"]
Sep 09 22:04:24.093350 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:04:24.093256    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf5039b1-bf1d-4743-808c-203161aee555" path="/var/lib/kubelet/pods/bf5039b1-bf1d-4743-808c-203161aee555/volumes"
Sep 09 22:05:13.192424 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192388    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"]
Sep 09 22:05:13.192963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192884    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ece9ec-7182-445e-bb2d-f2ee84df3577" containerName="payload-processing"
Sep 09 22:05:13.192963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192898    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ece9ec-7182-445e-bb2d-f2ee84df3577" containerName="payload-processing"
Sep 09 22:05:13.192963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192922    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy"
Sep 09 22:05:13.192963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192928    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy"
Sep 09 22:05:13.192963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192943    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0be3390-211e-4154-a690-b489ee4425c2" containerName="payload-pre-processing"
Sep 09 22:05:13.192963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192949    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0be3390-211e-4154-a690-b489ee4425c2" containerName="payload-pre-processing"
Sep 09 22:05:13.192963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192956    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5469077-ed75-445a-8a66-bad1c33d9b0f" containerName="revoke-keys"
Sep 09 22:05:13.192963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.192961    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5469077-ed75-445a-8a66-bad1c33d9b0f" containerName="revoke-keys"
Sep 09 22:05:13.193223 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.193019    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ece9ec-7182-445e-bb2d-f2ee84df3577" containerName="payload-processing"
Sep 09 22:05:13.193223 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.193026    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0be3390-211e-4154-a690-b489ee4425c2" containerName="payload-pre-processing"
Sep 09 22:05:13.193223 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.193032    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy"
Sep 09 22:05:13.193223 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.193038    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy"
Sep 09 22:05:13.193223 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.193044    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5469077-ed75-445a-8a66-bad1c33d9b0f" containerName="revoke-keys"
Sep 09 22:05:13.196509 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.196487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.199101 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.199074    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8a52f0-openshift-default-dockercfg-x2xwf\""
Sep 09 22:05:13.211647 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.211618    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"]
Sep 09 22:05:13.265740 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.265693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istiod-ca-cert\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.265959 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.265752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-data\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.265959 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.265839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9p8\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-kube-api-access-6z9p8\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.265959 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.265883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-socket\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.265959 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.265921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-envoy\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.266141 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.266022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-credential-socket\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.266141 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.266058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-certs\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.266141 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.266123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-podinfo\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.266240 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.266144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-token\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.366903 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.366863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-podinfo\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367118 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.366991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-token\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367118 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istiod-ca-cert\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367118 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-data\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367305 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9p8\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-kube-api-access-6z9p8\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367305 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-socket\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367426 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-envoy\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367480 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-credential-socket\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367537 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-certs\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367537 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-data\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367698 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-socket\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367863 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-credential-socket\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367933 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-certs\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.367933 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.367908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istiod-ca-cert\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.369478 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.369450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-envoy\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.369712 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.369695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-podinfo\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.375138 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.375115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-token\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.377698 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.375786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9p8\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-kube-api-access-6z9p8\") pod \"e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") " pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.507680 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.507564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:13.795245 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.795219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"]
Sep 09 22:05:13.796496 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:05:13.796471    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode926eb51_1d84_4a5a_aa04_bdc5789a6bfc.slice/crio-0c645e9525fcc218dbe8254b8cef12fe67a408aa28f011b5b74c71bb8861a7c4 WatchSource:0}: Error finding container 0c645e9525fcc218dbe8254b8cef12fe67a408aa28f011b5b74c71bb8861a7c4: Status 404 returned error can't find the container with id 0c645e9525fcc218dbe8254b8cef12fe67a408aa28f011b5b74c71bb8861a7c4
Sep 09 22:05:13.798782 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.798748    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:05:13.798860 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.798814    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:05:13.798860 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:13.798841    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:05:14.187473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:14.187443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" event={"ID":"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc","Type":"ContainerStarted","Data":"230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e"}
Sep 09 22:05:14.187695 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:14.187492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" event={"ID":"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc","Type":"ContainerStarted","Data":"0c645e9525fcc218dbe8254b8cef12fe67a408aa28f011b5b74c71bb8861a7c4"}
Sep 09 22:05:14.209432 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:14.209378    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podStartSLOduration=1.209362752 podStartE2EDuration="1.209362752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:14.208530083 +0000 UTC m=+1588.630892784" watchObservedRunningTime="2026-09-09 22:05:14.209362752 +0000 UTC m=+1588.631725451"
Sep 09 22:05:14.508003 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:14.507910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:14.513180 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:14.513149    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:15.193365 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:15.193336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:15.194536 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:15.194509    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:17.948132 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.948093    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"]
Sep 09 22:05:17.948798 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.948777    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy"
Sep 09 22:05:17.948921 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.948800    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5039b1-bf1d-4743-808c-203161aee555" containerName="istio-proxy"
Sep 09 22:05:17.951894 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.951872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:17.954727 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.954693    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8a52f0\""
Sep 09 22:05:17.954866 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.954845    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8a52f0-dockercfg-g9bfk\""
Sep 09 22:05:17.962061 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.962035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"]
Sep 09 22:05:17.983993 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.983965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"]
Sep 09 22:05:17.987474 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.987456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:17.998818 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:17.998789    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"]
Sep 09 22:05:18.015824 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.015791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r425q\" (UniqueName: \"kubernetes.io/projected/9117aeb5-237e-4ef1-94ca-807f10b49538-kube-api-access-r425q\") pod \"payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2\" (UID: \"9117aeb5-237e-4ef1-94ca-807f10b49538\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:18.016026 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.015945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9117aeb5-237e-4ef1-94ca-807f10b49538-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2\" (UID: \"9117aeb5-237e-4ef1-94ca-807f10b49538\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:18.116550 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.116516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r425q\" (UniqueName: \"kubernetes.io/projected/9117aeb5-237e-4ef1-94ca-807f10b49538-kube-api-access-r425q\") pod \"payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2\" (UID: \"9117aeb5-237e-4ef1-94ca-807f10b49538\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:18.116761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.116642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83c69b8-c860-4fae-9390-9a00bf16d03d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp\" (UID: \"c83c69b8-c860-4fae-9390-9a00bf16d03d\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:18.116761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.116725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9117aeb5-237e-4ef1-94ca-807f10b49538-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2\" (UID: \"9117aeb5-237e-4ef1-94ca-807f10b49538\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:18.116761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.116757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrmz\" (UniqueName: \"kubernetes.io/projected/c83c69b8-c860-4fae-9390-9a00bf16d03d-kube-api-access-jjrmz\") pod \"payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp\" (UID: \"c83c69b8-c860-4fae-9390-9a00bf16d03d\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:18.117386 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.117367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9117aeb5-237e-4ef1-94ca-807f10b49538-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2\" (UID: \"9117aeb5-237e-4ef1-94ca-807f10b49538\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:18.126269 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.126234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r425q\" (UniqueName: \"kubernetes.io/projected/9117aeb5-237e-4ef1-94ca-807f10b49538-kube-api-access-r425q\") pod \"payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2\" (UID: \"9117aeb5-237e-4ef1-94ca-807f10b49538\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:18.217598 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.217509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrmz\" (UniqueName: \"kubernetes.io/projected/c83c69b8-c860-4fae-9390-9a00bf16d03d-kube-api-access-jjrmz\") pod \"payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp\" (UID: \"c83c69b8-c860-4fae-9390-9a00bf16d03d\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:18.217783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.217729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83c69b8-c860-4fae-9390-9a00bf16d03d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp\" (UID: \"c83c69b8-c860-4fae-9390-9a00bf16d03d\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:18.218403 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.218377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83c69b8-c860-4fae-9390-9a00bf16d03d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp\" (UID: \"c83c69b8-c860-4fae-9390-9a00bf16d03d\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:18.226978 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.226956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrmz\" (UniqueName: \"kubernetes.io/projected/c83c69b8-c860-4fae-9390-9a00bf16d03d-kube-api-access-jjrmz\") pod \"payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp\" (UID: \"c83c69b8-c860-4fae-9390-9a00bf16d03d\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:18.263840 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.263806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:18.298082 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.298038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:18.540531 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.540500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"]
Sep 09 22:05:18.541585 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:05:18.541559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9117aeb5_237e_4ef1_94ca_807f10b49538.slice/crio-fe21515299eeb59482dc8a2921a85e5d1cbe47ac4d16b596ebb655c73490437b WatchSource:0}: Error finding container fe21515299eeb59482dc8a2921a85e5d1cbe47ac4d16b596ebb655c73490437b: Status 404 returned error can't find the container with id fe21515299eeb59482dc8a2921a85e5d1cbe47ac4d16b596ebb655c73490437b
Sep 09 22:05:18.586481 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:18.586453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"]
Sep 09 22:05:18.587380 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:05:18.587355    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83c69b8_c860_4fae_9390_9a00bf16d03d.slice/crio-aec1fefcf566f0c7d68293aa29b9677c921e294178a4dbd35377c1b2dd6c8553 WatchSource:0}: Error finding container aec1fefcf566f0c7d68293aa29b9677c921e294178a4dbd35377c1b2dd6c8553: Status 404 returned error can't find the container with id aec1fefcf566f0c7d68293aa29b9677c921e294178a4dbd35377c1b2dd6c8553
Sep 09 22:05:19.226478 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:19.226443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2" event={"ID":"9117aeb5-237e-4ef1-94ca-807f10b49538","Type":"ContainerStarted","Data":"a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084"}
Sep 09 22:05:19.226998 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:19.226499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2" event={"ID":"9117aeb5-237e-4ef1-94ca-807f10b49538","Type":"ContainerStarted","Data":"fe21515299eeb59482dc8a2921a85e5d1cbe47ac4d16b596ebb655c73490437b"}
Sep 09 22:05:19.226998 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:19.226573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:19.228241 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:19.228212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp" event={"ID":"c83c69b8-c860-4fae-9390-9a00bf16d03d","Type":"ContainerStarted","Data":"009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85"}
Sep 09 22:05:19.228379 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:19.228257    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:19.228379 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:19.228279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp" event={"ID":"c83c69b8-c860-4fae-9390-9a00bf16d03d","Type":"ContainerStarted","Data":"aec1fefcf566f0c7d68293aa29b9677c921e294178a4dbd35377c1b2dd6c8553"}
Sep 09 22:05:19.246321 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:19.246268    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2" podStartSLOduration=2.24625452 podStartE2EDuration="2.24625452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:19.244234229 +0000 UTC m=+1593.666596928" watchObservedRunningTime="2026-09-09 22:05:19.24625452 +0000 UTC m=+1593.668617219"
Sep 09 22:05:19.262775 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:19.262734    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp" podStartSLOduration=2.262721211 podStartE2EDuration="2.262721211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:19.260949107 +0000 UTC m=+1593.683311807" watchObservedRunningTime="2026-09-09 22:05:19.262721211 +0000 UTC m=+1593.685083912"
Sep 09 22:05:30.236112 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:30.236068    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:05:30.237343 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:30.237315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:05:31.234850 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:31.234796    2579 patch_prober.go:28] interesting pod/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 22:05:31.235026 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:31.234873    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 22:05:32.305978 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:32.305897    2579 generic.go:356] "Generic (PLEG): container finished" podID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerID="230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e" exitCode=0
Sep 09 22:05:32.306392 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:32.305973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" event={"ID":"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc","Type":"ContainerDied","Data":"230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e"}
Sep 09 22:05:32.306436 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:32.306424    2579 scope.go:117] "RemoveContainer" containerID="230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e"
Sep 09 22:05:32.307828 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:32.307790    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:05:32.307972 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:32.307878    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:05:32.307972 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:32.307912    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:05:33.326290 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:33.326253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" event={"ID":"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc","Type":"ContainerStarted","Data":"a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759"}
Sep 09 22:05:33.509616 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:33.509555    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:33.509851 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:33.509742    2579 patch_prober.go:28] interesting pod/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 22:05:33.509851 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:33.509802    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 22:05:34.508738 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:34.508692    2579 patch_prober.go:28] interesting pod/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 22:05:34.509349 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:34.509316    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 22:05:35.508216 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:35.508169    2579 patch_prober.go:28] interesting pod/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 22:05:35.508434 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:35.508239    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 22:05:36.524034 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:36.523999    2579 patch_prober.go:28] interesting pod/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 22:05:36.524829 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:36.524796    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 22:05:37.509015 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:37.508964    2579 patch_prober.go:28] interesting pod/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 22:05:37.509246 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:37.509057    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 22:05:38.512267 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:38.512169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:38.512780 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:38.512573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:38.513226 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:38.513207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:05:46.362446 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:46.362410    2579 scope.go:117] "RemoveContainer" containerID="01f75bcffb7e83f4dbd47ec6c5df2293c481a7a5529477cc9b98e1d51fe23000"
Sep 09 22:05:46.908791 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:46.908744    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"]
Sep 09 22:05:46.912887 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:46.912853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:46.915741 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:46.915714    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4f06aa\""
Sep 09 22:05:46.915884 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:46.915744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4f06aa-dockercfg-lk49s\""
Sep 09 22:05:46.922527 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:46.922506    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"]
Sep 09 22:05:47.052076 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:47.052038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8xz\" (UniqueName: \"kubernetes.io/projected/e92274b7-b6d3-4436-a905-0aeaea07929e-kube-api-access-lz8xz\") pod \"payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d\" (UID: \"e92274b7-b6d3-4436-a905-0aeaea07929e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:47.052250 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:47.052101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92274b7-b6d3-4436-a905-0aeaea07929e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d\" (UID: \"e92274b7-b6d3-4436-a905-0aeaea07929e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:47.153200 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:47.153152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8xz\" (UniqueName: \"kubernetes.io/projected/e92274b7-b6d3-4436-a905-0aeaea07929e-kube-api-access-lz8xz\") pod \"payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d\" (UID: \"e92274b7-b6d3-4436-a905-0aeaea07929e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:47.153400 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:47.153236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92274b7-b6d3-4436-a905-0aeaea07929e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d\" (UID: \"e92274b7-b6d3-4436-a905-0aeaea07929e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:47.154087 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:47.154064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92274b7-b6d3-4436-a905-0aeaea07929e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d\" (UID: \"e92274b7-b6d3-4436-a905-0aeaea07929e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:47.166052 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:47.165971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8xz\" (UniqueName: \"kubernetes.io/projected/e92274b7-b6d3-4436-a905-0aeaea07929e-kube-api-access-lz8xz\") pod \"payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d\" (UID: \"e92274b7-b6d3-4436-a905-0aeaea07929e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:47.225911 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:47.225877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:47.502834 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:47.502808    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"]
Sep 09 22:05:47.504961 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:05:47.504928    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92274b7_b6d3_4436_a905_0aeaea07929e.slice/crio-4eddf5e01fc15a61a8c9fae3d94e6e74e6d7432d2c0b3ef6e6a3becff2fdcf10 WatchSource:0}: Error finding container 4eddf5e01fc15a61a8c9fae3d94e6e74e6d7432d2c0b3ef6e6a3becff2fdcf10: Status 404 returned error can't find the container with id 4eddf5e01fc15a61a8c9fae3d94e6e74e6d7432d2c0b3ef6e6a3becff2fdcf10
Sep 09 22:05:48.431564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:48.431526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d" event={"ID":"e92274b7-b6d3-4436-a905-0aeaea07929e","Type":"ContainerStarted","Data":"ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b"}
Sep 09 22:05:48.431801 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:48.431629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d" event={"ID":"e92274b7-b6d3-4436-a905-0aeaea07929e","Type":"ContainerStarted","Data":"4eddf5e01fc15a61a8c9fae3d94e6e74e6d7432d2c0b3ef6e6a3becff2fdcf10"}
Sep 09 22:05:48.431801 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:48.431661    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:05:48.454161 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:48.454104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d" podStartSLOduration=2.4540845940000002 podStartE2EDuration="2.454084594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:05:48.451096045 +0000 UTC m=+1622.873458744" watchObservedRunningTime="2026-09-09 22:05:48.454084594 +0000 UTC m=+1622.876447296"
Sep 09 22:05:59.439100 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:05:59.439063    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:06:12.809585 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:12.809547    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"]
Sep 09 22:06:12.814895 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:12.814867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"
Sep 09 22:06:12.817829 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:12.817796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:06:12.820081 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:12.820056    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"]
Sep 09 22:06:12.923632 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:12.923563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4810207-5915-42cf-af8c-f24b48feca26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq\" (UID: \"f4810207-5915-42cf-af8c-f24b48feca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"
Sep 09 22:06:13.024675 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:13.024636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4810207-5915-42cf-af8c-f24b48feca26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq\" (UID: \"f4810207-5915-42cf-af8c-f24b48feca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"
Sep 09 22:06:13.025250 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:13.025230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4810207-5915-42cf-af8c-f24b48feca26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq\" (UID: \"f4810207-5915-42cf-af8c-f24b48feca26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"
Sep 09 22:06:13.128219 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:13.128174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"
Sep 09 22:06:13.401723 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:13.401670    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"]
Sep 09 22:06:13.402477 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:06:13.402453    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4810207_5915_42cf_af8c_f24b48feca26.slice/crio-239184f1818d057643e00f5400d3289e0fc6e65e795da69d34a263c607d31064 WatchSource:0}: Error finding container 239184f1818d057643e00f5400d3289e0fc6e65e795da69d34a263c607d31064: Status 404 returned error can't find the container with id 239184f1818d057643e00f5400d3289e0fc6e65e795da69d34a263c607d31064
Sep 09 22:06:13.582160 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:13.582117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq" event={"ID":"f4810207-5915-42cf-af8c-f24b48feca26","Type":"ContainerStarted","Data":"239184f1818d057643e00f5400d3289e0fc6e65e795da69d34a263c607d31064"}
Sep 09 22:06:14.589239 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:14.589154    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4810207-5915-42cf-af8c-f24b48feca26" containerID="5372062d0b5102eb7853873da52f474f8d19458e5f965d2365e31a6589fb135c" exitCode=0
Sep 09 22:06:14.589595 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:14.589237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq" event={"ID":"f4810207-5915-42cf-af8c-f24b48feca26","Type":"ContainerDied","Data":"5372062d0b5102eb7853873da52f474f8d19458e5f965d2365e31a6589fb135c"}
Sep 09 22:06:15.735994 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:15.735970    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"
Sep 09 22:06:15.854017 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:15.853990    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4810207-5915-42cf-af8c-f24b48feca26-maas-api-service-ca\") pod \"f4810207-5915-42cf-af8c-f24b48feca26\" (UID: \"f4810207-5915-42cf-af8c-f24b48feca26\") "
Sep 09 22:06:15.854349 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:15.854326    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4810207-5915-42cf-af8c-f24b48feca26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4810207-5915-42cf-af8c-f24b48feca26" (UID: "f4810207-5915-42cf-af8c-f24b48feca26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:15.955452 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:15.955421    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4810207-5915-42cf-af8c-f24b48feca26-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:16.601876 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:16.601843    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"
Sep 09 22:06:16.602088 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:16.601868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq" event={"ID":"f4810207-5915-42cf-af8c-f24b48feca26","Type":"ContainerDied","Data":"239184f1818d057643e00f5400d3289e0fc6e65e795da69d34a263c607d31064"}
Sep 09 22:06:16.602088 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:16.601929    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="239184f1818d057643e00f5400d3289e0fc6e65e795da69d34a263c607d31064"
Sep 09 22:06:28.593827 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.593792    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"]
Sep 09 22:06:28.594454 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.594090    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp" podUID="c83c69b8-c860-4fae-9390-9a00bf16d03d" containerName="payload-processing" containerID="cri-o://009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85" gracePeriod=30
Sep 09 22:06:28.599104 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.598971    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"]
Sep 09 22:06:28.599559 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.599307    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2" podUID="9117aeb5-237e-4ef1-94ca-807f10b49538" containerName="payload-pre-processing" containerID="cri-o://a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084" gracePeriod=30
Sep 09 22:06:28.773629 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.771262    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"]
Sep 09 22:06:28.778547 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.778484    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a52f0-xjftq"]
Sep 09 22:06:28.855092 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.855067    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:06:28.890760 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.890735    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:06:28.995696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.995659    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9117aeb5-237e-4ef1-94ca-807f10b49538-plugins-config-volume\") pod \"9117aeb5-237e-4ef1-94ca-807f10b49538\" (UID: \"9117aeb5-237e-4ef1-94ca-807f10b49538\") "
Sep 09 22:06:28.995893 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.995748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjrmz\" (UniqueName: \"kubernetes.io/projected/c83c69b8-c860-4fae-9390-9a00bf16d03d-kube-api-access-jjrmz\") pod \"c83c69b8-c860-4fae-9390-9a00bf16d03d\" (UID: \"c83c69b8-c860-4fae-9390-9a00bf16d03d\") "
Sep 09 22:06:28.995893 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.995864    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r425q\" (UniqueName: \"kubernetes.io/projected/9117aeb5-237e-4ef1-94ca-807f10b49538-kube-api-access-r425q\") pod \"9117aeb5-237e-4ef1-94ca-807f10b49538\" (UID: \"9117aeb5-237e-4ef1-94ca-807f10b49538\") "
Sep 09 22:06:28.996050 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.995900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83c69b8-c860-4fae-9390-9a00bf16d03d-plugins-config-volume\") pod \"c83c69b8-c860-4fae-9390-9a00bf16d03d\" (UID: \"c83c69b8-c860-4fae-9390-9a00bf16d03d\") "
Sep 09 22:06:28.996260 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.996131    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9117aeb5-237e-4ef1-94ca-807f10b49538-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9117aeb5-237e-4ef1-94ca-807f10b49538" (UID: "9117aeb5-237e-4ef1-94ca-807f10b49538"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:28.996348 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.996275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83c69b8-c860-4fae-9390-9a00bf16d03d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c83c69b8-c860-4fae-9390-9a00bf16d03d" (UID: "c83c69b8-c860-4fae-9390-9a00bf16d03d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:28.996411 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.996395    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9117aeb5-237e-4ef1-94ca-807f10b49538-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:28.996467 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.996412    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83c69b8-c860-4fae-9390-9a00bf16d03d-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:28.997950 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.997923    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9117aeb5-237e-4ef1-94ca-807f10b49538-kube-api-access-r425q" (OuterVolumeSpecName: "kube-api-access-r425q") pod "9117aeb5-237e-4ef1-94ca-807f10b49538" (UID: "9117aeb5-237e-4ef1-94ca-807f10b49538"). InnerVolumeSpecName "kube-api-access-r425q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:06:28.998042 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:28.997977    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83c69b8-c860-4fae-9390-9a00bf16d03d-kube-api-access-jjrmz" (OuterVolumeSpecName: "kube-api-access-jjrmz") pod "c83c69b8-c860-4fae-9390-9a00bf16d03d" (UID: "c83c69b8-c860-4fae-9390-9a00bf16d03d"). InnerVolumeSpecName "kube-api-access-jjrmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:06:29.097857 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.097819    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjrmz\" (UniqueName: \"kubernetes.io/projected/c83c69b8-c860-4fae-9390-9a00bf16d03d-kube-api-access-jjrmz\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:29.097857 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.097856    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r425q\" (UniqueName: \"kubernetes.io/projected/9117aeb5-237e-4ef1-94ca-807f10b49538-kube-api-access-r425q\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:29.684054 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.684016    2579 generic.go:356] "Generic (PLEG): container finished" podID="9117aeb5-237e-4ef1-94ca-807f10b49538" containerID="a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084" exitCode=0
Sep 09 22:06:29.684483 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.684065    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"
Sep 09 22:06:29.684483 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.684101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2" event={"ID":"9117aeb5-237e-4ef1-94ca-807f10b49538","Type":"ContainerDied","Data":"a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084"}
Sep 09 22:06:29.684483 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.684157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2" event={"ID":"9117aeb5-237e-4ef1-94ca-807f10b49538","Type":"ContainerDied","Data":"fe21515299eeb59482dc8a2921a85e5d1cbe47ac4d16b596ebb655c73490437b"}
Sep 09 22:06:29.684483 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.684184    2579 scope.go:117] "RemoveContainer" containerID="a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084"
Sep 09 22:06:29.685651 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.685626    2579 generic.go:356] "Generic (PLEG): container finished" podID="c83c69b8-c860-4fae-9390-9a00bf16d03d" containerID="009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85" exitCode=0
Sep 09 22:06:29.685745 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.685678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"
Sep 09 22:06:29.685745 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.685717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp" event={"ID":"c83c69b8-c860-4fae-9390-9a00bf16d03d","Type":"ContainerDied","Data":"009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85"}
Sep 09 22:06:29.685858 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.685762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp" event={"ID":"c83c69b8-c860-4fae-9390-9a00bf16d03d","Type":"ContainerDied","Data":"aec1fefcf566f0c7d68293aa29b9677c921e294178a4dbd35377c1b2dd6c8553"}
Sep 09 22:06:29.694653 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.694632    2579 scope.go:117] "RemoveContainer" containerID="a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084"
Sep 09 22:06:29.695077 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:06:29.695049    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084\": container with ID starting with a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084 not found: ID does not exist" containerID="a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084"
Sep 09 22:06:29.695198 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.695098    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084"} err="failed to get container status \"a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084\": rpc error: code = NotFound desc = could not find container \"a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084\": container with ID starting with a40e2c0ef72528497dfc570ec19e229ce9be4761183576a6d71bccfb76521084 not found: ID does not exist"
Sep 09 22:06:29.695198 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.695118    2579 scope.go:117] "RemoveContainer" containerID="009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85"
Sep 09 22:06:29.705569 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.705551    2579 scope.go:117] "RemoveContainer" containerID="009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85"
Sep 09 22:06:29.705924 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:06:29.705903    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85\": container with ID starting with 009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85 not found: ID does not exist" containerID="009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85"
Sep 09 22:06:29.705990 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.705932    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85"} err="failed to get container status \"009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85\": rpc error: code = NotFound desc = could not find container \"009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85\": container with ID starting with 009787c43224a04c5b2ad9de7c258afc72f9318e15b7fa07f95be26fd3afad85 not found: ID does not exist"
Sep 09 22:06:29.727177 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.727150    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"]
Sep 09 22:06:29.729366 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.729344    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a52f0-79f574b4f5-9b2mp"]
Sep 09 22:06:29.748845 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.748818    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"]
Sep 09 22:06:29.755661 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:29.755639    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a52f0-764fd8bf78-drqh2"]
Sep 09 22:06:30.092761 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:30.092678    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9117aeb5-237e-4ef1-94ca-807f10b49538" path="/var/lib/kubelet/pods/9117aeb5-237e-4ef1-94ca-807f10b49538/volumes"
Sep 09 22:06:30.093140 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:30.093125    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83c69b8-c860-4fae-9390-9a00bf16d03d" path="/var/lib/kubelet/pods/c83c69b8-c860-4fae-9390-9a00bf16d03d/volumes"
Sep 09 22:06:30.093466 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:30.093453    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4810207-5915-42cf-af8c-f24b48feca26" path="/var/lib/kubelet/pods/f4810207-5915-42cf-af8c-f24b48feca26/volumes"
Sep 09 22:06:35.120644 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.120593    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"]
Sep 09 22:06:35.121167 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.120879    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy" containerID="cri-o://a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759" gracePeriod=30
Sep 09 22:06:35.466848 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.466745    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"]
Sep 09 22:06:35.467282 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467265    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4810207-5915-42cf-af8c-f24b48feca26" containerName="revoke-keys"
Sep 09 22:06:35.467336 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467285    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4810207-5915-42cf-af8c-f24b48feca26" containerName="revoke-keys"
Sep 09 22:06:35.467336 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467296    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9117aeb5-237e-4ef1-94ca-807f10b49538" containerName="payload-pre-processing"
Sep 09 22:06:35.467336 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467304    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9117aeb5-237e-4ef1-94ca-807f10b49538" containerName="payload-pre-processing"
Sep 09 22:06:35.467336 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467317    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83c69b8-c860-4fae-9390-9a00bf16d03d" containerName="payload-processing"
Sep 09 22:06:35.467336 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467323    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83c69b8-c860-4fae-9390-9a00bf16d03d" containerName="payload-processing"
Sep 09 22:06:35.467502 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467401    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9117aeb5-237e-4ef1-94ca-807f10b49538" containerName="payload-pre-processing"
Sep 09 22:06:35.467502 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467411    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83c69b8-c860-4fae-9390-9a00bf16d03d" containerName="payload-processing"
Sep 09 22:06:35.467502 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.467418    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4810207-5915-42cf-af8c-f24b48feca26" containerName="revoke-keys"
Sep 09 22:06:35.472068 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.472050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"
Sep 09 22:06:35.474714 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.474683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:06:35.478136 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.478107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"]
Sep 09 22:06:35.561913 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.561872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45bce79-079f-42da-a932-da507e745893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9\" (UID: \"e45bce79-079f-42da-a932-da507e745893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"
Sep 09 22:06:35.663250 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.663217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45bce79-079f-42da-a932-da507e745893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9\" (UID: \"e45bce79-079f-42da-a932-da507e745893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"
Sep 09 22:06:35.663879 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.663857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45bce79-079f-42da-a932-da507e745893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9\" (UID: \"e45bce79-079f-42da-a932-da507e745893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"
Sep 09 22:06:35.784348 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:35.784251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"
Sep 09 22:06:36.055476 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:36.055450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"]
Sep 09 22:06:36.057393 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:06:36.057367    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45bce79_079f_42da_a932_da507e745893.slice/crio-461c5fd70a16dd6c3e4aa1a77fc0fddd4962b775303afe4a6b11a5b489744598 WatchSource:0}: Error finding container 461c5fd70a16dd6c3e4aa1a77fc0fddd4962b775303afe4a6b11a5b489744598: Status 404 returned error can't find the container with id 461c5fd70a16dd6c3e4aa1a77fc0fddd4962b775303afe4a6b11a5b489744598
Sep 09 22:06:36.733047 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:36.733013    2579 generic.go:356] "Generic (PLEG): container finished" podID="e45bce79-079f-42da-a932-da507e745893" containerID="67a3dc334e598c503342a54c4fce8d72764acde1151c04832a4d84c9bc2cd056" exitCode=0
Sep 09 22:06:36.733433 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:36.733107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9" event={"ID":"e45bce79-079f-42da-a932-da507e745893","Type":"ContainerDied","Data":"67a3dc334e598c503342a54c4fce8d72764acde1151c04832a4d84c9bc2cd056"}
Sep 09 22:06:36.733433 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:36.733161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9" event={"ID":"e45bce79-079f-42da-a932-da507e745893","Type":"ContainerStarted","Data":"461c5fd70a16dd6c3e4aa1a77fc0fddd4962b775303afe4a6b11a5b489744598"}
Sep 09 22:06:37.878100 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:37.878076    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"
Sep 09 22:06:37.989802 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:37.989712    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45bce79-079f-42da-a932-da507e745893-maas-api-service-ca\") pod \"e45bce79-079f-42da-a932-da507e745893\" (UID: \"e45bce79-079f-42da-a932-da507e745893\") "
Sep 09 22:06:37.990095 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:37.990069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45bce79-079f-42da-a932-da507e745893-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e45bce79-079f-42da-a932-da507e745893" (UID: "e45bce79-079f-42da-a932-da507e745893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:38.090850 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:38.090813    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45bce79-079f-42da-a932-da507e745893-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:38.515629 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:38.515560    2579 patch_prober.go:28] interesting pod/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 22:06:38.515809 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:38.515643    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 22:06:38.751451 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:38.751415    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"
Sep 09 22:06:38.751451 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:38.751444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9" event={"ID":"e45bce79-079f-42da-a932-da507e745893","Type":"ContainerDied","Data":"461c5fd70a16dd6c3e4aa1a77fc0fddd4962b775303afe4a6b11a5b489744598"}
Sep 09 22:06:38.751673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:38.751513    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="461c5fd70a16dd6c3e4aa1a77fc0fddd4962b775303afe4a6b11a5b489744598"
Sep 09 22:06:40.375861 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.375836    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:06:40.413251 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413163    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-data\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413251 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-token\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-socket\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413309    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z9p8\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-kube-api-access-6z9p8\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-credential-socket\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413443    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-data" (OuterVolumeSpecName: "istio-data") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:06:40.413712 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413481    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-podinfo\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413712 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413484    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:06:40.413712 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istiod-ca-cert\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413712 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-envoy\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413712 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413589    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-certs\") pod \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\" (UID: \"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc\") "
Sep 09 22:06:40.413963 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413725    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:06:40.414023 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.413958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:06:40.414100 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.414070    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.414100 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.414096    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.414218 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.414109    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.414395 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.414314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:40.414395 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.414352    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.416162 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.416138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 22:06:40.416634 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.416578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:06:40.416634 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.416615    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-kube-api-access-6z9p8" (OuterVolumeSpecName: "kube-api-access-6z9p8") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "kube-api-access-6z9p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:06:40.416634 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.416630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-token" (OuterVolumeSpecName: "istio-token") pod "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" (UID: "e926eb51-1d84-4a5a-aa04-bdc5789a6bfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:06:40.515263 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.515229    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.515263 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.515259    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9p8\" (UniqueName: \"kubernetes.io/projected/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-kube-api-access-6z9p8\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.515454 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.515270    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.515454 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.515287    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.515454 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.515296    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:40.764535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.764438    2579 generic.go:356] "Generic (PLEG): container finished" podID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerID="a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759" exitCode=0
Sep 09 22:06:40.764535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.764486    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"
Sep 09 22:06:40.764535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.764509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" event={"ID":"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc","Type":"ContainerDied","Data":"a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759"}
Sep 09 22:06:40.764768 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.764555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628" event={"ID":"e926eb51-1d84-4a5a-aa04-bdc5789a6bfc","Type":"ContainerDied","Data":"0c645e9525fcc218dbe8254b8cef12fe67a408aa28f011b5b74c71bb8861a7c4"}
Sep 09 22:06:40.764768 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.764578    2579 scope.go:117] "RemoveContainer" containerID="a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759"
Sep 09 22:06:40.775148 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.775129    2579 scope.go:117] "RemoveContainer" containerID="230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e"
Sep 09 22:06:40.783687 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.783669    2579 scope.go:117] "RemoveContainer" containerID="a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759"
Sep 09 22:06:40.784463 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:06:40.784079    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759\": container with ID starting with a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759 not found: ID does not exist" containerID="a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759"
Sep 09 22:06:40.784463 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.784119    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759"} err="failed to get container status \"a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759\": rpc error: code = NotFound desc = could not find container \"a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759\": container with ID starting with a865992c82b13342ee7317102519c5072553fb5c5d6c99545bcf3c6f5551d759 not found: ID does not exist"
Sep 09 22:06:40.784463 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.784144    2579 scope.go:117] "RemoveContainer" containerID="230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e"
Sep 09 22:06:40.784731 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:06:40.784645    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e\": container with ID starting with 230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e not found: ID does not exist" containerID="230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e"
Sep 09 22:06:40.784731 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.784678    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e"} err="failed to get container status \"230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e\": rpc error: code = NotFound desc = could not find container \"230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e\": container with ID starting with 230072592ba25b394fdc5d2bbea5858819a7afb0968aade4c115a13be2861c4e not found: ID does not exist"
Sep 09 22:06:40.787155 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.787136    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"]
Sep 09 22:06:40.792680 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:40.792657    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8a52f0-openshift-default-5bdfd6975b-wm628"]
Sep 09 22:06:42.092977 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:42.092937    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" path="/var/lib/kubelet/pods/e926eb51-1d84-4a5a-aa04-bdc5789a6bfc/volumes"
Sep 09 22:06:46.385115 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:46.385082    2579 scope.go:117] "RemoveContainer" containerID="291a8fb78db0a0ba42f8eb1f55ff1e39d4c36b16b0967b1e04b82aff4579ef43"
Sep 09 22:06:51.130339 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.130303    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"]
Sep 09 22:06:51.130879 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.130540    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d" podUID="e92274b7-b6d3-4436-a905-0aeaea07929e" containerName="payload-processing" containerID="cri-o://ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b" gracePeriod=30
Sep 09 22:06:51.295831 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.295795    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"]
Sep 09 22:06:51.299075 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.299045    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f06aa-bc8d9"]
Sep 09 22:06:51.380986 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.380839    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:06:51.523983 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.523948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz8xz\" (UniqueName: \"kubernetes.io/projected/e92274b7-b6d3-4436-a905-0aeaea07929e-kube-api-access-lz8xz\") pod \"e92274b7-b6d3-4436-a905-0aeaea07929e\" (UID: \"e92274b7-b6d3-4436-a905-0aeaea07929e\") "
Sep 09 22:06:51.524156 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.524005    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92274b7-b6d3-4436-a905-0aeaea07929e-plugins-config-volume\") pod \"e92274b7-b6d3-4436-a905-0aeaea07929e\" (UID: \"e92274b7-b6d3-4436-a905-0aeaea07929e\") "
Sep 09 22:06:51.524419 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.524393    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92274b7-b6d3-4436-a905-0aeaea07929e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e92274b7-b6d3-4436-a905-0aeaea07929e" (UID: "e92274b7-b6d3-4436-a905-0aeaea07929e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:06:51.526107 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.526082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92274b7-b6d3-4436-a905-0aeaea07929e-kube-api-access-lz8xz" (OuterVolumeSpecName: "kube-api-access-lz8xz") pod "e92274b7-b6d3-4436-a905-0aeaea07929e" (UID: "e92274b7-b6d3-4436-a905-0aeaea07929e"). InnerVolumeSpecName "kube-api-access-lz8xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:06:51.625022 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.624987    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8xz\" (UniqueName: \"kubernetes.io/projected/e92274b7-b6d3-4436-a905-0aeaea07929e-kube-api-access-lz8xz\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:51.625022 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.625017    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92274b7-b6d3-4436-a905-0aeaea07929e-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:06:51.828198 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.828116    2579 generic.go:356] "Generic (PLEG): container finished" podID="e92274b7-b6d3-4436-a905-0aeaea07929e" containerID="ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b" exitCode=0
Sep 09 22:06:51.828198 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.828161    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"
Sep 09 22:06:51.828388 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.828197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d" event={"ID":"e92274b7-b6d3-4436-a905-0aeaea07929e","Type":"ContainerDied","Data":"ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b"}
Sep 09 22:06:51.828388 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.828250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d" event={"ID":"e92274b7-b6d3-4436-a905-0aeaea07929e","Type":"ContainerDied","Data":"4eddf5e01fc15a61a8c9fae3d94e6e74e6d7432d2c0b3ef6e6a3becff2fdcf10"}
Sep 09 22:06:51.828388 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.828273    2579 scope.go:117] "RemoveContainer" containerID="ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b"
Sep 09 22:06:51.837150 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.837134    2579 scope.go:117] "RemoveContainer" containerID="ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b"
Sep 09 22:06:51.837412 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:06:51.837392    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b\": container with ID starting with ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b not found: ID does not exist" containerID="ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b"
Sep 09 22:06:51.837454 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.837420    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b"} err="failed to get container status \"ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b\": rpc error: code = NotFound desc = could not find container \"ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b\": container with ID starting with ca96cd861b469a6aba0891800328862067e533f5a3cfb6de4c14eceedc95351b not found: ID does not exist"
Sep 09 22:06:51.851779 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.851757    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"]
Sep 09 22:06:51.856556 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:51.856538    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f06aa-5cd7579bb9-jf62d"]
Sep 09 22:06:52.093049 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:52.092967    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e45bce79-079f-42da-a932-da507e745893" path="/var/lib/kubelet/pods/e45bce79-079f-42da-a932-da507e745893/volumes"
Sep 09 22:06:52.093356 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:52.093343    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92274b7-b6d3-4436-a905-0aeaea07929e" path="/var/lib/kubelet/pods/e92274b7-b6d3-4436-a905-0aeaea07929e/volumes"
Sep 09 22:06:58.299692 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.299643    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"]
Sep 09 22:06:58.300125 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300109    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy"
Sep 09 22:06:58.300125 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300126    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy"
Sep 09 22:06:58.300203 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300137    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy"
Sep 09 22:06:58.300203 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300143    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy"
Sep 09 22:06:58.300203 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300166    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92274b7-b6d3-4436-a905-0aeaea07929e" containerName="payload-processing"
Sep 09 22:06:58.300203 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300172    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92274b7-b6d3-4436-a905-0aeaea07929e" containerName="payload-processing"
Sep 09 22:06:58.300203 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300178    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e45bce79-079f-42da-a932-da507e745893" containerName="revoke-keys"
Sep 09 22:06:58.300203 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300183    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e45bce79-079f-42da-a932-da507e745893" containerName="revoke-keys"
Sep 09 22:06:58.300380 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300248    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e45bce79-079f-42da-a932-da507e745893" containerName="revoke-keys"
Sep 09 22:06:58.300380 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300255    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy"
Sep 09 22:06:58.300380 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300261    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92274b7-b6d3-4436-a905-0aeaea07929e" containerName="payload-processing"
Sep 09 22:06:58.300380 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.300267    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e926eb51-1d84-4a5a-aa04-bdc5789a6bfc" containerName="istio-proxy"
Sep 09 22:06:58.304805 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.304785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"
Sep 09 22:06:58.307352 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.307328    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:06:58.311629 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.311584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"]
Sep 09 22:06:58.392149 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.392108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e911955-adaa-4841-af36-203d47c2895c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr\" (UID: \"4e911955-adaa-4841-af36-203d47c2895c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"
Sep 09 22:06:58.442130 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.442096    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"]
Sep 09 22:06:58.445838 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.445813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.448636 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.448593    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2f2a52-openshift-default-dockercfg-8kvdw\""
Sep 09 22:06:58.457927 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.457899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"]
Sep 09 22:06:58.493320 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.493292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e911955-adaa-4841-af36-203d47c2895c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr\" (UID: \"4e911955-adaa-4841-af36-203d47c2895c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"
Sep 09 22:06:58.494014 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.493977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e911955-adaa-4841-af36-203d47c2895c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr\" (UID: \"4e911955-adaa-4841-af36-203d47c2895c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"
Sep 09 22:06:58.594654 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-podinfo\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.594654 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-socket\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.594654 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-certs\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.594878 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954vq\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-kube-api-access-954vq\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.594878 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451ec2a6-f3b9-4079-9dec-d92c10c89990-istiod-ca-cert\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.594878 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-credential-socket\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.594878 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-data\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.595006 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-token\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.595006 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.594914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-envoy\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.616349 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.616323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"
Sep 09 22:06:58.696638 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451ec2a6-f3b9-4079-9dec-d92c10c89990-istiod-ca-cert\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.696638 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-credential-socket\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.696638 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-data\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.696853 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-token\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.696853 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-envoy\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.696853 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-podinfo\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.696853 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-socket\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.696853 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-certs\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.697021 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-954vq\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-kube-api-access-954vq\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.697021 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.696971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-credential-socket\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.697687 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.697291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-socket\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.697687 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.697304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451ec2a6-f3b9-4079-9dec-d92c10c89990-istiod-ca-cert\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.697687 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.697529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-certs\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.698321 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.698295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-data\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.699724 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.699699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-envoy\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.700385 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.700353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-podinfo\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.709673 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.709651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-954vq\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-kube-api-access-954vq\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.711404 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.711381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-token\") pod \"e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") " pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.757639 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.757585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:06:58.886858 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:58.886832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"]
Sep 09 22:06:58.888684 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:06:58.888654    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e911955_adaa_4841_af36_203d47c2895c.slice/crio-f539ff0c680c145b9a19cab6bc976928b47dc1424bb2d4cabfb906e36d987ba2 WatchSource:0}: Error finding container f539ff0c680c145b9a19cab6bc976928b47dc1424bb2d4cabfb906e36d987ba2: Status 404 returned error can't find the container with id f539ff0c680c145b9a19cab6bc976928b47dc1424bb2d4cabfb906e36d987ba2
Sep 09 22:06:59.035516 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.035418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"]
Sep 09 22:06:59.037885 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:06:59.037843    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451ec2a6_f3b9_4079_9dec_d92c10c89990.slice/crio-6b0c8fc79fbad5d162197662850a7f365568b559071a481823bfa0ea1c924832 WatchSource:0}: Error finding container 6b0c8fc79fbad5d162197662850a7f365568b559071a481823bfa0ea1c924832: Status 404 returned error can't find the container with id 6b0c8fc79fbad5d162197662850a7f365568b559071a481823bfa0ea1c924832
Sep 09 22:06:59.040371 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.040337    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:06:59.040439 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.040413    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:06:59.040479 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.040442    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:06:59.878774 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.878737    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e911955-adaa-4841-af36-203d47c2895c" containerID="f04641200e40d5d05fe38cf7f3fa1b9e1dbd6cc7d374da4177ffb90c94eed5aa" exitCode=0
Sep 09 22:06:59.879264 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.878823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr" event={"ID":"4e911955-adaa-4841-af36-203d47c2895c","Type":"ContainerDied","Data":"f04641200e40d5d05fe38cf7f3fa1b9e1dbd6cc7d374da4177ffb90c94eed5aa"}
Sep 09 22:06:59.879264 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.878895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr" event={"ID":"4e911955-adaa-4841-af36-203d47c2895c","Type":"ContainerStarted","Data":"f539ff0c680c145b9a19cab6bc976928b47dc1424bb2d4cabfb906e36d987ba2"}
Sep 09 22:06:59.880661 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.880636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" event={"ID":"451ec2a6-f3b9-4079-9dec-d92c10c89990","Type":"ContainerStarted","Data":"a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480"}
Sep 09 22:06:59.880785 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.880683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" event={"ID":"451ec2a6-f3b9-4079-9dec-d92c10c89990","Type":"ContainerStarted","Data":"6b0c8fc79fbad5d162197662850a7f365568b559071a481823bfa0ea1c924832"}
Sep 09 22:06:59.921764 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:06:59.921702    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" podStartSLOduration=1.9216841489999998 podStartE2EDuration="1.921684149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:06:59.919221716 +0000 UTC m=+1694.341584415" watchObservedRunningTime="2026-09-09 22:06:59.921684149 +0000 UTC m=+1694.344046849"
Sep 09 22:07:00.758442 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:00.758403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:07:00.763514 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:00.763488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:07:00.886535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:00.886493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:07:00.887588 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:00.887564    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:07:01.049248 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:01.049148    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"
Sep 09 22:07:01.220671 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:01.220631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e911955-adaa-4841-af36-203d47c2895c-maas-api-service-ca\") pod \"4e911955-adaa-4841-af36-203d47c2895c\" (UID: \"4e911955-adaa-4841-af36-203d47c2895c\") "
Sep 09 22:07:01.220975 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:01.220953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e911955-adaa-4841-af36-203d47c2895c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e911955-adaa-4841-af36-203d47c2895c" (UID: "4e911955-adaa-4841-af36-203d47c2895c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:01.321870 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:01.321777    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e911955-adaa-4841-af36-203d47c2895c-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:07:01.893182 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:01.893152    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"
Sep 09 22:07:01.893630 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:01.893188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr" event={"ID":"4e911955-adaa-4841-af36-203d47c2895c","Type":"ContainerDied","Data":"f539ff0c680c145b9a19cab6bc976928b47dc1424bb2d4cabfb906e36d987ba2"}
Sep 09 22:07:01.893630 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:01.893234    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f539ff0c680c145b9a19cab6bc976928b47dc1424bb2d4cabfb906e36d987ba2"
Sep 09 22:07:03.606473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.606435    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"]
Sep 09 22:07:03.606981 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.606965    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e911955-adaa-4841-af36-203d47c2895c" containerName="revoke-keys"
Sep 09 22:07:03.607034 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.606985    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e911955-adaa-4841-af36-203d47c2895c" containerName="revoke-keys"
Sep 09 22:07:03.607074 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.607063    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e911955-adaa-4841-af36-203d47c2895c" containerName="revoke-keys"
Sep 09 22:07:03.610382 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.610368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:03.614443 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.614419    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2f2a52-dockercfg-5hz7l\""
Sep 09 22:07:03.614839 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.614821    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2f2a52\""
Sep 09 22:07:03.626118 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.626092    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"]
Sep 09 22:07:03.749327 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.749289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwh98\" (UniqueName: \"kubernetes.io/projected/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-kube-api-access-hwh98\") pod \"payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb\" (UID: \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:03.749512 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.749338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb\" (UID: \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:03.850043 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.850009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwh98\" (UniqueName: \"kubernetes.io/projected/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-kube-api-access-hwh98\") pod \"payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb\" (UID: \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:03.850225 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.850054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb\" (UID: \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:03.850839 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.850816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb\" (UID: \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:03.861694 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.861628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwh98\" (UniqueName: \"kubernetes.io/projected/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-kube-api-access-hwh98\") pod \"payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb\" (UID: \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:03.921204 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:03.921173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:04.206734 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:04.206702    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"]
Sep 09 22:07:04.208960 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:07:04.208934    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6435b1f3_8fb2_42cc_a9f0_9fa7178403c9.slice/crio-5bf8e4520e085eadb6637b153019aeedc77687e9b79168c49c34e4edec5284ff WatchSource:0}: Error finding container 5bf8e4520e085eadb6637b153019aeedc77687e9b79168c49c34e4edec5284ff: Status 404 returned error can't find the container with id 5bf8e4520e085eadb6637b153019aeedc77687e9b79168c49c34e4edec5284ff
Sep 09 22:07:04.914009 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:04.913969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb" event={"ID":"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9","Type":"ContainerStarted","Data":"51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc"}
Sep 09 22:07:04.914433 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:04.914031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb" event={"ID":"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9","Type":"ContainerStarted","Data":"5bf8e4520e085eadb6637b153019aeedc77687e9b79168c49c34e4edec5284ff"}
Sep 09 22:07:04.914433 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:04.914061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:04.942110 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:04.942051    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb" podStartSLOduration=1.942032691 podStartE2EDuration="1.942032691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:04.931204598 +0000 UTC m=+1699.353567301" watchObservedRunningTime="2026-09-09 22:07:04.942032691 +0000 UTC m=+1699.364395391"
Sep 09 22:07:09.079251 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.079162    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"]
Sep 09 22:07:09.079809 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.079430    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh" podUID="22e6ae76-f05a-4af3-be25-8c32d0d45a7c" containerName="payload-processing" containerID="cri-o://21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef" gracePeriod=30
Sep 09 22:07:09.090037 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.089803    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"]
Sep 09 22:07:09.090037 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.090022    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df" podUID="01acd97a-bcae-4005-be37-764a83892734" containerName="payload-pre-processing" containerID="cri-o://a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c" gracePeriod=30
Sep 09 22:07:09.263180 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.263151    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"]
Sep 09 22:07:09.266513 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.266487    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6cce2-6grhr"]
Sep 09 22:07:09.377847 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.377818    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:07:09.380773 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.380747    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:07:09.408269 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.408239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcxk8\" (UniqueName: \"kubernetes.io/projected/01acd97a-bcae-4005-be37-764a83892734-kube-api-access-gcxk8\") pod \"01acd97a-bcae-4005-be37-764a83892734\" (UID: \"01acd97a-bcae-4005-be37-764a83892734\") "
Sep 09 22:07:09.408269 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.408275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5qbj\" (UniqueName: \"kubernetes.io/projected/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-kube-api-access-d5qbj\") pod \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\" (UID: \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\") "
Sep 09 22:07:09.408481 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.408296    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-plugins-config-volume\") pod \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\" (UID: \"22e6ae76-f05a-4af3-be25-8c32d0d45a7c\") "
Sep 09 22:07:09.408481 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.408438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01acd97a-bcae-4005-be37-764a83892734-plugins-config-volume\") pod \"01acd97a-bcae-4005-be37-764a83892734\" (UID: \"01acd97a-bcae-4005-be37-764a83892734\") "
Sep 09 22:07:09.408843 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.408786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22e6ae76-f05a-4af3-be25-8c32d0d45a7c" (UID: "22e6ae76-f05a-4af3-be25-8c32d0d45a7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:09.408991 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.408849    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01acd97a-bcae-4005-be37-764a83892734-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01acd97a-bcae-4005-be37-764a83892734" (UID: "01acd97a-bcae-4005-be37-764a83892734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:09.408991 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.408940    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:07:09.410474 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.410448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-kube-api-access-d5qbj" (OuterVolumeSpecName: "kube-api-access-d5qbj") pod "22e6ae76-f05a-4af3-be25-8c32d0d45a7c" (UID: "22e6ae76-f05a-4af3-be25-8c32d0d45a7c"). InnerVolumeSpecName "kube-api-access-d5qbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:09.410867 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.410848    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01acd97a-bcae-4005-be37-764a83892734-kube-api-access-gcxk8" (OuterVolumeSpecName: "kube-api-access-gcxk8") pod "01acd97a-bcae-4005-be37-764a83892734" (UID: "01acd97a-bcae-4005-be37-764a83892734"). InnerVolumeSpecName "kube-api-access-gcxk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:09.509994 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.509962    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcxk8\" (UniqueName: \"kubernetes.io/projected/01acd97a-bcae-4005-be37-764a83892734-kube-api-access-gcxk8\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:07:09.509994 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.509990    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qbj\" (UniqueName: \"kubernetes.io/projected/22e6ae76-f05a-4af3-be25-8c32d0d45a7c-kube-api-access-d5qbj\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:07:09.509994 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.510000    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01acd97a-bcae-4005-be37-764a83892734-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:07:09.944766 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.944733    2579 generic.go:356] "Generic (PLEG): container finished" podID="01acd97a-bcae-4005-be37-764a83892734" containerID="a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c" exitCode=0
Sep 09 22:07:09.944988 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.944775    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"
Sep 09 22:07:09.944988 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.944820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df" event={"ID":"01acd97a-bcae-4005-be37-764a83892734","Type":"ContainerDied","Data":"a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c"}
Sep 09 22:07:09.944988 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.944882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df" event={"ID":"01acd97a-bcae-4005-be37-764a83892734","Type":"ContainerDied","Data":"bccde30f50e4f0e72210e46a6d2c26bccc5db4f06c770fb42c3a36a3e635d6be"}
Sep 09 22:07:09.944988 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.944917    2579 scope.go:117] "RemoveContainer" containerID="a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c"
Sep 09 22:07:09.946433 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.946411    2579 generic.go:356] "Generic (PLEG): container finished" podID="22e6ae76-f05a-4af3-be25-8c32d0d45a7c" containerID="21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef" exitCode=0
Sep 09 22:07:09.946542 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.946454    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"
Sep 09 22:07:09.946542 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.946481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh" event={"ID":"22e6ae76-f05a-4af3-be25-8c32d0d45a7c","Type":"ContainerDied","Data":"21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef"}
Sep 09 22:07:09.946542 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.946523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh" event={"ID":"22e6ae76-f05a-4af3-be25-8c32d0d45a7c","Type":"ContainerDied","Data":"a290016284635b5d429bab0ed14f71f54fa30208ed2e57952907660a1a69fa9b"}
Sep 09 22:07:09.955082 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.955059    2579 scope.go:117] "RemoveContainer" containerID="a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c"
Sep 09 22:07:09.955451 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:07:09.955422    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c\": container with ID starting with a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c not found: ID does not exist" containerID="a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c"
Sep 09 22:07:09.955567 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.955456    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c"} err="failed to get container status \"a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c\": rpc error: code = NotFound desc = could not find container \"a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c\": container with ID starting with a950f8607970c951e1feaf53ab45681069390bf689d4d0dae11ab12936cc2e1c not found: ID does not exist"
Sep 09 22:07:09.955567 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.955474    2579 scope.go:117] "RemoveContainer" containerID="21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef"
Sep 09 22:07:09.964257 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.964235    2579 scope.go:117] "RemoveContainer" containerID="21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef"
Sep 09 22:07:09.964523 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:07:09.964504    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef\": container with ID starting with 21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef not found: ID does not exist" containerID="21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef"
Sep 09 22:07:09.964571 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.964530    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef"} err="failed to get container status \"21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef\": rpc error: code = NotFound desc = could not find container \"21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef\": container with ID starting with 21e7462a99f3cb8329dcbe1d63f5c2f3c3b9997ba387a2fa660ef24d57bef9ef not found: ID does not exist"
Sep 09 22:07:09.971255 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.971233    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"]
Sep 09 22:07:09.976127 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.976104    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6cce2-95bbb6458-vg6df"]
Sep 09 22:07:09.995634 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.988402    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"]
Sep 09 22:07:09.998229 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:09.998201    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6cce2-f4849b779-mtxxh"]
Sep 09 22:07:10.093717 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:10.093681    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01acd97a-bcae-4005-be37-764a83892734" path="/var/lib/kubelet/pods/01acd97a-bcae-4005-be37-764a83892734/volumes"
Sep 09 22:07:10.094211 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:10.094158    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e6ae76-f05a-4af3-be25-8c32d0d45a7c" path="/var/lib/kubelet/pods/22e6ae76-f05a-4af3-be25-8c32d0d45a7c/volumes"
Sep 09 22:07:10.094471 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:10.094454    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e911955-adaa-4841-af36-203d47c2895c" path="/var/lib/kubelet/pods/4e911955-adaa-4841-af36-203d47c2895c/volumes"
Sep 09 22:07:15.922780 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:15.922745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:07:16.106701 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.106670    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"]
Sep 09 22:07:16.107115 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.107101    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e6ae76-f05a-4af3-be25-8c32d0d45a7c" containerName="payload-processing"
Sep 09 22:07:16.107161 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.107116    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e6ae76-f05a-4af3-be25-8c32d0d45a7c" containerName="payload-processing"
Sep 09 22:07:16.107161 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.107131    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01acd97a-bcae-4005-be37-764a83892734" containerName="payload-pre-processing"
Sep 09 22:07:16.107161 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.107138    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="01acd97a-bcae-4005-be37-764a83892734" containerName="payload-pre-processing"
Sep 09 22:07:16.107262 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.107195    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e6ae76-f05a-4af3-be25-8c32d0d45a7c" containerName="payload-processing"
Sep 09 22:07:16.107262 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.107205    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="01acd97a-bcae-4005-be37-764a83892734" containerName="payload-pre-processing"
Sep 09 22:07:16.110339 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.110322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"
Sep 09 22:07:16.113134 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.113112    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:07:16.129632 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.129593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"]
Sep 09 22:07:16.181263 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.181165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17e6104-2b3e-4ad8-a231-f8cc696fcac6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x\" (UID: \"d17e6104-2b3e-4ad8-a231-f8cc696fcac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"
Sep 09 22:07:16.283455 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.283342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17e6104-2b3e-4ad8-a231-f8cc696fcac6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x\" (UID: \"d17e6104-2b3e-4ad8-a231-f8cc696fcac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"
Sep 09 22:07:16.284576 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.284542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17e6104-2b3e-4ad8-a231-f8cc696fcac6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x\" (UID: \"d17e6104-2b3e-4ad8-a231-f8cc696fcac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"
Sep 09 22:07:16.421680 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.421644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"
Sep 09 22:07:16.714260 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.714229    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"]
Sep 09 22:07:16.716084 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:07:16.716049    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17e6104_2b3e_4ad8_a231_f8cc696fcac6.slice/crio-5ebcd38be3fcf0c2ca2f1fc21f2b2bbdbe9267004b3a333f437b1212c5a055f4 WatchSource:0}: Error finding container 5ebcd38be3fcf0c2ca2f1fc21f2b2bbdbe9267004b3a333f437b1212c5a055f4: Status 404 returned error can't find the container with id 5ebcd38be3fcf0c2ca2f1fc21f2b2bbdbe9267004b3a333f437b1212c5a055f4
Sep 09 22:07:16.992414 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.992331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x" event={"ID":"d17e6104-2b3e-4ad8-a231-f8cc696fcac6","Type":"ContainerStarted","Data":"f53d0f5e61d83b9cb4664af15a260b563e79972f1503f69388ac506825a99075"}
Sep 09 22:07:16.992414 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:16.992389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x" event={"ID":"d17e6104-2b3e-4ad8-a231-f8cc696fcac6","Type":"ContainerStarted","Data":"5ebcd38be3fcf0c2ca2f1fc21f2b2bbdbe9267004b3a333f437b1212c5a055f4"}
Sep 09 22:07:17.015641 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:17.015568    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x" podStartSLOduration=1.015550473 podStartE2EDuration="1.015550473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:17.013799009 +0000 UTC m=+1711.436161710" watchObservedRunningTime="2026-09-09 22:07:17.015550473 +0000 UTC m=+1711.437913175"
Sep 09 22:07:17.999536 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:17.999494    2579 generic.go:356] "Generic (PLEG): container finished" podID="d17e6104-2b3e-4ad8-a231-f8cc696fcac6" containerID="f53d0f5e61d83b9cb4664af15a260b563e79972f1503f69388ac506825a99075" exitCode=0
Sep 09 22:07:17.999955 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:17.999580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x" event={"ID":"d17e6104-2b3e-4ad8-a231-f8cc696fcac6","Type":"ContainerDied","Data":"f53d0f5e61d83b9cb4664af15a260b563e79972f1503f69388ac506825a99075"}
Sep 09 22:07:19.007581 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.007557    2579 generic.go:356] "Generic (PLEG): container finished" podID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerID="a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480" exitCode=0
Sep 09 22:07:19.008010 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.007637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" event={"ID":"451ec2a6-f3b9-4079-9dec-d92c10c89990","Type":"ContainerDied","Data":"a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480"}
Sep 09 22:07:19.008174 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.008153    2579 scope.go:117] "RemoveContainer" containerID="a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480"
Sep 09 22:07:19.009346 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.009315    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:07:19.009429 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.009388    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:07:19.009429 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.009423    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 09 22:07:19.175519 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.175482    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"
Sep 09 22:07:19.215668 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.215633    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17e6104-2b3e-4ad8-a231-f8cc696fcac6-maas-api-service-ca\") pod \"d17e6104-2b3e-4ad8-a231-f8cc696fcac6\" (UID: \"d17e6104-2b3e-4ad8-a231-f8cc696fcac6\") "
Sep 09 22:07:19.216040 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.216016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17e6104-2b3e-4ad8-a231-f8cc696fcac6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d17e6104-2b3e-4ad8-a231-f8cc696fcac6" (UID: "d17e6104-2b3e-4ad8-a231-f8cc696fcac6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:19.316841 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:19.316759    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17e6104-2b3e-4ad8-a231-f8cc696fcac6-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:07:20.026949 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:20.026916    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"
Sep 09 22:07:20.027477 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:20.026973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x" event={"ID":"d17e6104-2b3e-4ad8-a231-f8cc696fcac6","Type":"ContainerDied","Data":"5ebcd38be3fcf0c2ca2f1fc21f2b2bbdbe9267004b3a333f437b1212c5a055f4"}
Sep 09 22:07:20.027477 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:20.027019    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ebcd38be3fcf0c2ca2f1fc21f2b2bbdbe9267004b3a333f437b1212c5a055f4"
Sep 09 22:07:20.029674 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:20.029645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" event={"ID":"451ec2a6-f3b9-4079-9dec-d92c10c89990","Type":"ContainerStarted","Data":"7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701"}
Sep 09 22:07:20.758449 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:20.758406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:07:20.758681 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:20.758659    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 22:07:20.758737 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:20.758717    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 22:07:21.758583 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:21.758547    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 22:07:21.759037 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:21.758627    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 22:07:22.758153 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:22.758116    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 22:07:22.758341 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:22.758184    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 22:07:23.758240 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:23.758206    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 22:07:23.758689 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:23.758259    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 22:07:24.758691 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:24.758655    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 09 22:07:24.759076 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:24.758710    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 09 22:07:25.865467 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:25.865431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:07:25.865896 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:25.865738    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:07:25.866559 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:25.866536    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:07:26.361654 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.361599    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"]
Sep 09 22:07:26.361922 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.361890    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms" podUID="07c538dd-d704-4390-a26f-bdefe2203253" containerName="payload-processing" containerID="cri-o://8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad" gracePeriod=30
Sep 09 22:07:26.580616 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.580558    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"]
Sep 09 22:07:26.585791 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.585760    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e5cb43-pq57x"]
Sep 09 22:07:26.650075 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.650051    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:07:26.811157 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.811120    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54dvz\" (UniqueName: \"kubernetes.io/projected/07c538dd-d704-4390-a26f-bdefe2203253-kube-api-access-54dvz\") pod \"07c538dd-d704-4390-a26f-bdefe2203253\" (UID: \"07c538dd-d704-4390-a26f-bdefe2203253\") "
Sep 09 22:07:26.811344 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.811196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c538dd-d704-4390-a26f-bdefe2203253-plugins-config-volume\") pod \"07c538dd-d704-4390-a26f-bdefe2203253\" (UID: \"07c538dd-d704-4390-a26f-bdefe2203253\") "
Sep 09 22:07:26.811660 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.811627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c538dd-d704-4390-a26f-bdefe2203253-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07c538dd-d704-4390-a26f-bdefe2203253" (UID: "07c538dd-d704-4390-a26f-bdefe2203253"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:07:26.813224 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.813205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c538dd-d704-4390-a26f-bdefe2203253-kube-api-access-54dvz" (OuterVolumeSpecName: "kube-api-access-54dvz") pod "07c538dd-d704-4390-a26f-bdefe2203253" (UID: "07c538dd-d704-4390-a26f-bdefe2203253"). InnerVolumeSpecName "kube-api-access-54dvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:26.912115 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.912031    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54dvz\" (UniqueName: \"kubernetes.io/projected/07c538dd-d704-4390-a26f-bdefe2203253-kube-api-access-54dvz\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:07:26.912115 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:26.912062    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07c538dd-d704-4390-a26f-bdefe2203253-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:07:27.070893 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.070858    2579 generic.go:356] "Generic (PLEG): container finished" podID="07c538dd-d704-4390-a26f-bdefe2203253" containerID="8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad" exitCode=0
Sep 09 22:07:27.071059 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.070912    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"
Sep 09 22:07:27.071059 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.070940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms" event={"ID":"07c538dd-d704-4390-a26f-bdefe2203253","Type":"ContainerDied","Data":"8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad"}
Sep 09 22:07:27.071059 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.071022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms" event={"ID":"07c538dd-d704-4390-a26f-bdefe2203253","Type":"ContainerDied","Data":"c3fe0181af8702605d1829820c835a07e458348af8e46ff8e91c8358b7564eda"}
Sep 09 22:07:27.071059 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.071048    2579 scope.go:117] "RemoveContainer" containerID="8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad"
Sep 09 22:07:27.080506 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.080490    2579 scope.go:117] "RemoveContainer" containerID="8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad"
Sep 09 22:07:27.080787 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:07:27.080764    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad\": container with ID starting with 8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad not found: ID does not exist" containerID="8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad"
Sep 09 22:07:27.080871 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.080798    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad"} err="failed to get container status \"8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad\": rpc error: code = NotFound desc = could not find container \"8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad\": container with ID starting with 8d86fc02e7171e2248d841ca5caf3472b5f3e29762ab86545e993b976ccda1ad not found: ID does not exist"
Sep 09 22:07:27.101561 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.101533    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"]
Sep 09 22:07:27.110228 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:27.110201    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e5cb43-7f69fb4655-8pwms"]
Sep 09 22:07:28.092518 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:28.092484    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c538dd-d704-4390-a26f-bdefe2203253" path="/var/lib/kubelet/pods/07c538dd-d704-4390-a26f-bdefe2203253/volumes"
Sep 09 22:07:28.092932 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:28.092864    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17e6104-2b3e-4ad8-a231-f8cc696fcac6" path="/var/lib/kubelet/pods/d17e6104-2b3e-4ad8-a231-f8cc696fcac6/volumes"
Sep 09 22:07:32.324083 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.324043    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"]
Sep 09 22:07:32.324543 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.324528    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17e6104-2b3e-4ad8-a231-f8cc696fcac6" containerName="revoke-keys"
Sep 09 22:07:32.324543 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.324545    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17e6104-2b3e-4ad8-a231-f8cc696fcac6" containerName="revoke-keys"
Sep 09 22:07:32.324665 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.324573    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c538dd-d704-4390-a26f-bdefe2203253" containerName="payload-processing"
Sep 09 22:07:32.324665 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.324582    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c538dd-d704-4390-a26f-bdefe2203253" containerName="payload-processing"
Sep 09 22:07:32.324738 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.324665    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17e6104-2b3e-4ad8-a231-f8cc696fcac6" containerName="revoke-keys"
Sep 09 22:07:32.324738 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.324677    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c538dd-d704-4390-a26f-bdefe2203253" containerName="payload-processing"
Sep 09 22:07:32.329085 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.329069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:32.332102 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.332081    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8fe6a1-dockercfg-jdxcm\""
Sep 09 22:07:32.332254 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.332183    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8fe6a1\""
Sep 09 22:07:32.339279 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.339259    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"]
Sep 09 22:07:32.342792 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.342772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"]
Sep 09 22:07:32.342892 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.342875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:32.359734 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.359713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"]
Sep 09 22:07:32.471282 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.471244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlqd\" (UniqueName: \"kubernetes.io/projected/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-kube-api-access-2tlqd\") pod \"payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l\" (UID: \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:32.471496 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.471308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5kd\" (UniqueName: \"kubernetes.io/projected/d304d3a8-fb0f-451a-af89-1090dabf9136-kube-api-access-qq5kd\") pod \"payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp\" (UID: \"d304d3a8-fb0f-451a-af89-1090dabf9136\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:32.471496 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.471346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d304d3a8-fb0f-451a-af89-1090dabf9136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp\" (UID: \"d304d3a8-fb0f-451a-af89-1090dabf9136\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:32.471496 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.471378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l\" (UID: \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:32.572373 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.572328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlqd\" (UniqueName: \"kubernetes.io/projected/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-kube-api-access-2tlqd\") pod \"payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l\" (UID: \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:32.572535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.572394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5kd\" (UniqueName: \"kubernetes.io/projected/d304d3a8-fb0f-451a-af89-1090dabf9136-kube-api-access-qq5kd\") pod \"payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp\" (UID: \"d304d3a8-fb0f-451a-af89-1090dabf9136\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:32.572535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.572426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d304d3a8-fb0f-451a-af89-1090dabf9136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp\" (UID: \"d304d3a8-fb0f-451a-af89-1090dabf9136\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:32.572535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.572465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l\" (UID: \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:32.573425 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.573399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d304d3a8-fb0f-451a-af89-1090dabf9136-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp\" (UID: \"d304d3a8-fb0f-451a-af89-1090dabf9136\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:32.573477 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.573399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l\" (UID: \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:32.580960 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.580884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5kd\" (UniqueName: \"kubernetes.io/projected/d304d3a8-fb0f-451a-af89-1090dabf9136-kube-api-access-qq5kd\") pod \"payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp\" (UID: \"d304d3a8-fb0f-451a-af89-1090dabf9136\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:32.580960 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.580901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlqd\" (UniqueName: \"kubernetes.io/projected/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-kube-api-access-2tlqd\") pod \"payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l\" (UID: \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:32.640504 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.640454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:32.652836 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.652801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:32.921800 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.921774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"]
Sep 09 22:07:32.922943 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:07:32.922912    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16dbbdb_21a0_4ed3_a0c4_3dd96d52d2da.slice/crio-fdbddd76ab460516bf6b3857b6ef5925c24336df6f9f7e984eaf21339da41a0d WatchSource:0}: Error finding container fdbddd76ab460516bf6b3857b6ef5925c24336df6f9f7e984eaf21339da41a0d: Status 404 returned error can't find the container with id fdbddd76ab460516bf6b3857b6ef5925c24336df6f9f7e984eaf21339da41a0d
Sep 09 22:07:32.942276 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:32.942248    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"]
Sep 09 22:07:32.943388 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:07:32.943364    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd304d3a8_fb0f_451a_af89_1090dabf9136.slice/crio-310df171ae3f3e8e3e54e9f7301713afa33c14ba995e16a94a53355c36f43af6 WatchSource:0}: Error finding container 310df171ae3f3e8e3e54e9f7301713afa33c14ba995e16a94a53355c36f43af6: Status 404 returned error can't find the container with id 310df171ae3f3e8e3e54e9f7301713afa33c14ba995e16a94a53355c36f43af6
Sep 09 22:07:33.114714 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:33.114674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp" event={"ID":"d304d3a8-fb0f-451a-af89-1090dabf9136","Type":"ContainerStarted","Data":"139a1584b854c8f06fac2a5b10ae4ebcab564af84c7c1f427f6f0968e416ff8a"}
Sep 09 22:07:33.114917 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:33.114739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp" event={"ID":"d304d3a8-fb0f-451a-af89-1090dabf9136","Type":"ContainerStarted","Data":"310df171ae3f3e8e3e54e9f7301713afa33c14ba995e16a94a53355c36f43af6"}
Sep 09 22:07:33.114917 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:33.114821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:33.116486 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:33.116464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l" event={"ID":"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da","Type":"ContainerStarted","Data":"064b8bbab3098e27e44f970903048e2b57756ef35d43d6ba8cde6a4e69476f2b"}
Sep 09 22:07:33.116648 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:33.116515    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:07:33.116648 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:33.116540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l" event={"ID":"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da","Type":"ContainerStarted","Data":"fdbddd76ab460516bf6b3857b6ef5925c24336df6f9f7e984eaf21339da41a0d"}
Sep 09 22:07:33.135231 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:33.135176    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp" podStartSLOduration=1.135156812 podStartE2EDuration="1.135156812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:33.132352251 +0000 UTC m=+1727.554714964" watchObservedRunningTime="2026-09-09 22:07:33.135156812 +0000 UTC m=+1727.557519516"
Sep 09 22:07:33.154944 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:33.154896    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l" podStartSLOduration=1.154877356 podStartE2EDuration="1.154877356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:33.153391678 +0000 UTC m=+1727.575754381" watchObservedRunningTime="2026-09-09 22:07:33.154877356 +0000 UTC m=+1727.577240058"
Sep 09 22:07:44.126774 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:44.126740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:07:44.127277 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:07:44.126826    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:08:00.658900 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:00.658864    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp_d304d3a8-fb0f-451a-af89-1090dabf9136/payload-processing/0.log"
Sep 09 22:08:36.566794 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:36.566765    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp_d304d3a8-fb0f-451a-af89-1090dabf9136/payload-processing/0.log"
Sep 09 22:08:41.791287 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.791198    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"]
Sep 09 22:08:41.795295 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.795275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:41.797885 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.797867    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-fcfdbf\""
Sep 09 22:08:41.802561 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.802539    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"]
Sep 09 22:08:41.806282 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.806263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:41.810656 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.810632    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"]
Sep 09 22:08:41.811099 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.811080    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-fcfdbf-dockercfg-fljqd\""
Sep 09 22:08:41.821863 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.821830    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"]
Sep 09 22:08:41.985621 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.985570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13563f7-de2c-4902-b8a5-5ef5391d158c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl\" (UID: \"a13563f7-de2c-4902-b8a5-5ef5391d158c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:41.985828 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.985664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g\" (UID: \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:41.985828 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.985762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnnpt\" (UniqueName: \"kubernetes.io/projected/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-kube-api-access-pnnpt\") pod \"payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g\" (UID: \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:41.985960 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:41.985831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnh4t\" (UniqueName: \"kubernetes.io/projected/a13563f7-de2c-4902-b8a5-5ef5391d158c-kube-api-access-bnh4t\") pod \"payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl\" (UID: \"a13563f7-de2c-4902-b8a5-5ef5391d158c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:42.086587 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.086490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnnpt\" (UniqueName: \"kubernetes.io/projected/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-kube-api-access-pnnpt\") pod \"payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g\" (UID: \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:42.086587 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.086583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnh4t\" (UniqueName: \"kubernetes.io/projected/a13563f7-de2c-4902-b8a5-5ef5391d158c-kube-api-access-bnh4t\") pod \"payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl\" (UID: \"a13563f7-de2c-4902-b8a5-5ef5391d158c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:42.086864 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.086684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13563f7-de2c-4902-b8a5-5ef5391d158c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl\" (UID: \"a13563f7-de2c-4902-b8a5-5ef5391d158c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:42.086936 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.086857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g\" (UID: \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:42.087473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.087456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g\" (UID: \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:42.087525 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.087504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13563f7-de2c-4902-b8a5-5ef5391d158c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl\" (UID: \"a13563f7-de2c-4902-b8a5-5ef5391d158c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:42.098813 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.098785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnh4t\" (UniqueName: \"kubernetes.io/projected/a13563f7-de2c-4902-b8a5-5ef5391d158c-kube-api-access-bnh4t\") pod \"payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl\" (UID: \"a13563f7-de2c-4902-b8a5-5ef5391d158c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:42.098966 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.098848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnnpt\" (UniqueName: \"kubernetes.io/projected/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-kube-api-access-pnnpt\") pod \"payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g\" (UID: \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:42.107558 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.107535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:42.119650 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.119629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:42.385240 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.385210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"]
Sep 09 22:08:42.386861 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:08:42.386831    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13563f7_de2c_4902_b8a5_5ef5391d158c.slice/crio-eb23c4b238110a84eb24d477c549b85da7b8d5b4fa3ece544d92af26fc135616 WatchSource:0}: Error finding container eb23c4b238110a84eb24d477c549b85da7b8d5b4fa3ece544d92af26fc135616: Status 404 returned error can't find the container with id eb23c4b238110a84eb24d477c549b85da7b8d5b4fa3ece544d92af26fc135616
Sep 09 22:08:42.406901 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.406866    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"]
Sep 09 22:08:42.408838 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:08:42.408813    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a45d201_a7a6_4e76_b6d0_79ac71a983b5.slice/crio-31f7fa929e994fd8b020333a83f0c125b4943725f11b70867e5a4d1ef1a96fa2 WatchSource:0}: Error finding container 31f7fa929e994fd8b020333a83f0c125b4943725f11b70867e5a4d1ef1a96fa2: Status 404 returned error can't find the container with id 31f7fa929e994fd8b020333a83f0c125b4943725f11b70867e5a4d1ef1a96fa2
Sep 09 22:08:42.538433 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.538392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g" event={"ID":"5a45d201-a7a6-4e76-b6d0-79ac71a983b5","Type":"ContainerStarted","Data":"a8b69b5cfbc7eb79a7913fe2012fe3888e832c4a342631d23ea0da0361b3e540"}
Sep 09 22:08:42.538653 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.538458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g" event={"ID":"5a45d201-a7a6-4e76-b6d0-79ac71a983b5","Type":"ContainerStarted","Data":"31f7fa929e994fd8b020333a83f0c125b4943725f11b70867e5a4d1ef1a96fa2"}
Sep 09 22:08:42.538653 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.538587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:08:42.540243 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.540216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl" event={"ID":"a13563f7-de2c-4902-b8a5-5ef5391d158c","Type":"ContainerStarted","Data":"766954131d9f067eaefbd734b063fc7728485fe61266f523a7b387491887884c"}
Sep 09 22:08:42.540368 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.540264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:42.540368 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.540289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl" event={"ID":"a13563f7-de2c-4902-b8a5-5ef5391d158c","Type":"ContainerStarted","Data":"eb23c4b238110a84eb24d477c549b85da7b8d5b4fa3ece544d92af26fc135616"}
Sep 09 22:08:42.555619 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.555565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g" podStartSLOduration=1.555550784 podStartE2EDuration="1.555550784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:08:42.553798931 +0000 UTC m=+1796.976161646" watchObservedRunningTime="2026-09-09 22:08:42.555550784 +0000 UTC m=+1796.977913486"
Sep 09 22:08:42.570785 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:42.570743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl" podStartSLOduration=1.570730179 podStartE2EDuration="1.570730179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:08:42.569058005 +0000 UTC m=+1796.991420705" watchObservedRunningTime="2026-09-09 22:08:42.570730179 +0000 UTC m=+1796.993092878"
Sep 09 22:08:46.444349 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:46.444316    2579 scope.go:117] "RemoveContainer" containerID="40cba332ec672d2042a0f5fa85f644d9cabffc400bc7c0ace0bbc3d72af76a45"
Sep 09 22:08:53.550656 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:53.550624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:08:53.551065 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:08:53.550817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:09:05.558490 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:05.558456    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"]
Sep 09 22:09:05.562240 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:05.562224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"
Sep 09 22:09:05.569039 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:05.568800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:09:05.590999 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:05.590971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"]
Sep 09 22:09:05.633430 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:05.633396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c56f1-0bb0-48bc-a7fb-5daec275dc5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6\" (UID: \"663c56f1-0bb0-48bc-a7fb-5daec275dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"
Sep 09 22:09:05.734880 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:05.734844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c56f1-0bb0-48bc-a7fb-5daec275dc5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6\" (UID: \"663c56f1-0bb0-48bc-a7fb-5daec275dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"
Sep 09 22:09:05.735451 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:05.735431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c56f1-0bb0-48bc-a7fb-5daec275dc5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6\" (UID: \"663c56f1-0bb0-48bc-a7fb-5daec275dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"
Sep 09 22:09:05.873050 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:05.873020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"
Sep 09 22:09:06.145284 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:06.145258    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"]
Sep 09 22:09:06.146915 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:09:06.146890    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663c56f1_0bb0_48bc_a7fb_5daec275dc5a.slice/crio-b36bbf4c6cd5a50a57879942bf97e45366b0501c2fe282e34396e2a2697bbd2d WatchSource:0}: Error finding container b36bbf4c6cd5a50a57879942bf97e45366b0501c2fe282e34396e2a2697bbd2d: Status 404 returned error can't find the container with id b36bbf4c6cd5a50a57879942bf97e45366b0501c2fe282e34396e2a2697bbd2d
Sep 09 22:09:06.691916 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:06.691827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6" event={"ID":"663c56f1-0bb0-48bc-a7fb-5daec275dc5a","Type":"ContainerStarted","Data":"b53e203b7a44652efad34ab837356aded2d51233d94eed5c602ef28373d600d1"}
Sep 09 22:09:06.691916 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:06.691892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6" event={"ID":"663c56f1-0bb0-48bc-a7fb-5daec275dc5a","Type":"ContainerStarted","Data":"b36bbf4c6cd5a50a57879942bf97e45366b0501c2fe282e34396e2a2697bbd2d"}
Sep 09 22:09:06.709368 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:06.709316    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6" podStartSLOduration=1.7092990129999999 podStartE2EDuration="1.709299013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:09:06.707809095 +0000 UTC m=+1821.130171798" watchObservedRunningTime="2026-09-09 22:09:06.709299013 +0000 UTC m=+1821.131661712"
Sep 09 22:09:07.698449 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:07.698416    2579 generic.go:356] "Generic (PLEG): container finished" podID="663c56f1-0bb0-48bc-a7fb-5daec275dc5a" containerID="b53e203b7a44652efad34ab837356aded2d51233d94eed5c602ef28373d600d1" exitCode=0
Sep 09 22:09:07.698939 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:07.698494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6" event={"ID":"663c56f1-0bb0-48bc-a7fb-5daec275dc5a","Type":"ContainerDied","Data":"b53e203b7a44652efad34ab837356aded2d51233d94eed5c602ef28373d600d1"}
Sep 09 22:09:08.844045 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:08.844012    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"
Sep 09 22:09:08.966056 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:08.965975    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c56f1-0bb0-48bc-a7fb-5daec275dc5a-maas-api-service-ca\") pod \"663c56f1-0bb0-48bc-a7fb-5daec275dc5a\" (UID: \"663c56f1-0bb0-48bc-a7fb-5daec275dc5a\") "
Sep 09 22:09:08.966340 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:08.966316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663c56f1-0bb0-48bc-a7fb-5daec275dc5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "663c56f1-0bb0-48bc-a7fb-5daec275dc5a" (UID: "663c56f1-0bb0-48bc-a7fb-5daec275dc5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:09.067449 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:09.067410    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663c56f1-0bb0-48bc-a7fb-5daec275dc5a-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:09:09.710355 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:09.710324    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"
Sep 09 22:09:09.710542 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:09.710370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6" event={"ID":"663c56f1-0bb0-48bc-a7fb-5daec275dc5a","Type":"ContainerDied","Data":"b36bbf4c6cd5a50a57879942bf97e45366b0501c2fe282e34396e2a2697bbd2d"}
Sep 09 22:09:09.710542 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:09.710415    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b36bbf4c6cd5a50a57879942bf97e45366b0501c2fe282e34396e2a2697bbd2d"
Sep 09 22:09:11.554723 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.554692    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"]
Sep 09 22:09:11.555240 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.554933    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g" podUID="5a45d201-a7a6-4e76-b6d0-79ac71a983b5" containerName="payload-processing" containerID="cri-o://a8b69b5cfbc7eb79a7913fe2012fe3888e832c4a342631d23ea0da0361b3e540" gracePeriod=30
Sep 09 22:09:11.560892 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.560867    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"]
Sep 09 22:09:11.561230 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.561198    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl" podUID="a13563f7-de2c-4902-b8a5-5ef5391d158c" containerName="payload-pre-processing" containerID="cri-o://766954131d9f067eaefbd734b063fc7728485fe61266f523a7b387491887884c" gracePeriod=30
Sep 09 22:09:11.726667 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.726629    2579 generic.go:356] "Generic (PLEG): container finished" podID="5a45d201-a7a6-4e76-b6d0-79ac71a983b5" containerID="a8b69b5cfbc7eb79a7913fe2012fe3888e832c4a342631d23ea0da0361b3e540" exitCode=0
Sep 09 22:09:11.726970 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.726695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g" event={"ID":"5a45d201-a7a6-4e76-b6d0-79ac71a983b5","Type":"ContainerDied","Data":"a8b69b5cfbc7eb79a7913fe2012fe3888e832c4a342631d23ea0da0361b3e540"}
Sep 09 22:09:11.729133 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.729104    2579 generic.go:356] "Generic (PLEG): container finished" podID="a13563f7-de2c-4902-b8a5-5ef5391d158c" containerID="766954131d9f067eaefbd734b063fc7728485fe61266f523a7b387491887884c" exitCode=0
Sep 09 22:09:11.729278 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.729175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl" event={"ID":"a13563f7-de2c-4902-b8a5-5ef5391d158c","Type":"ContainerDied","Data":"766954131d9f067eaefbd734b063fc7728485fe61266f523a7b387491887884c"}
Sep 09 22:09:11.744076 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.743878    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"]
Sep 09 22:09:11.746836 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.746795    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fcfdbf-2v4w6"]
Sep 09 22:09:11.827017 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.826990    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:09:11.860255 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:11.860231    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:09:12.001188 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.001150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13563f7-de2c-4902-b8a5-5ef5391d158c-plugins-config-volume\") pod \"a13563f7-de2c-4902-b8a5-5ef5391d158c\" (UID: \"a13563f7-de2c-4902-b8a5-5ef5391d158c\") "
Sep 09 22:09:12.001378 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.001261    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnnpt\" (UniqueName: \"kubernetes.io/projected/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-kube-api-access-pnnpt\") pod \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\" (UID: \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\") "
Sep 09 22:09:12.001378 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.001287    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-plugins-config-volume\") pod \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\" (UID: \"5a45d201-a7a6-4e76-b6d0-79ac71a983b5\") "
Sep 09 22:09:12.001378 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.001314    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnh4t\" (UniqueName: \"kubernetes.io/projected/a13563f7-de2c-4902-b8a5-5ef5391d158c-kube-api-access-bnh4t\") pod \"a13563f7-de2c-4902-b8a5-5ef5391d158c\" (UID: \"a13563f7-de2c-4902-b8a5-5ef5391d158c\") "
Sep 09 22:09:12.001653 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.001597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13563f7-de2c-4902-b8a5-5ef5391d158c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a13563f7-de2c-4902-b8a5-5ef5391d158c" (UID: "a13563f7-de2c-4902-b8a5-5ef5391d158c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:12.001788 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.001755    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a45d201-a7a6-4e76-b6d0-79ac71a983b5" (UID: "5a45d201-a7a6-4e76-b6d0-79ac71a983b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:12.003570 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.003538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13563f7-de2c-4902-b8a5-5ef5391d158c-kube-api-access-bnh4t" (OuterVolumeSpecName: "kube-api-access-bnh4t") pod "a13563f7-de2c-4902-b8a5-5ef5391d158c" (UID: "a13563f7-de2c-4902-b8a5-5ef5391d158c"). InnerVolumeSpecName "kube-api-access-bnh4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:12.003719 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.003562    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-kube-api-access-pnnpt" (OuterVolumeSpecName: "kube-api-access-pnnpt") pod "5a45d201-a7a6-4e76-b6d0-79ac71a983b5" (UID: "5a45d201-a7a6-4e76-b6d0-79ac71a983b5"). InnerVolumeSpecName "kube-api-access-pnnpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:12.092591 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.092492    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="663c56f1-0bb0-48bc-a7fb-5daec275dc5a" path="/var/lib/kubelet/pods/663c56f1-0bb0-48bc-a7fb-5daec275dc5a/volumes"
Sep 09 22:09:12.102420 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.102391    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnnpt\" (UniqueName: \"kubernetes.io/projected/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-kube-api-access-pnnpt\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:09:12.102537 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.102426    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a45d201-a7a6-4e76-b6d0-79ac71a983b5-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:09:12.102537 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.102443    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnh4t\" (UniqueName: \"kubernetes.io/projected/a13563f7-de2c-4902-b8a5-5ef5391d158c-kube-api-access-bnh4t\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:09:12.102537 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.102460    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13563f7-de2c-4902-b8a5-5ef5391d158c-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:09:12.736495 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.736462    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"
Sep 09 22:09:12.736971 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.736494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g" event={"ID":"5a45d201-a7a6-4e76-b6d0-79ac71a983b5","Type":"ContainerDied","Data":"31f7fa929e994fd8b020333a83f0c125b4943725f11b70867e5a4d1ef1a96fa2"}
Sep 09 22:09:12.736971 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.736577    2579 scope.go:117] "RemoveContainer" containerID="a8b69b5cfbc7eb79a7913fe2012fe3888e832c4a342631d23ea0da0361b3e540"
Sep 09 22:09:12.738181 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.738161    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"
Sep 09 22:09:12.738319 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.738218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl" event={"ID":"a13563f7-de2c-4902-b8a5-5ef5391d158c","Type":"ContainerDied","Data":"eb23c4b238110a84eb24d477c549b85da7b8d5b4fa3ece544d92af26fc135616"}
Sep 09 22:09:12.746428 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.746410    2579 scope.go:117] "RemoveContainer" containerID="766954131d9f067eaefbd734b063fc7728485fe61266f523a7b387491887884c"
Sep 09 22:09:12.757316 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.757293    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"]
Sep 09 22:09:12.761006 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.760983    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fcfdbf-9456554cd-tfrwl"]
Sep 09 22:09:12.773969 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.773949    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"]
Sep 09 22:09:12.779454 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:12.779423    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fcfdbf-68f7d48b8f-t742g"]
Sep 09 22:09:14.094102 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:14.094068    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a45d201-a7a6-4e76-b6d0-79ac71a983b5" path="/var/lib/kubelet/pods/5a45d201-a7a6-4e76-b6d0-79ac71a983b5/volumes"
Sep 09 22:09:14.094492 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:14.094456    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a13563f7-de2c-4902-b8a5-5ef5391d158c" path="/var/lib/kubelet/pods/a13563f7-de2c-4902-b8a5-5ef5391d158c/volumes"
Sep 09 22:09:21.785100 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785062    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"]
Sep 09 22:09:21.785676 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785657    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a13563f7-de2c-4902-b8a5-5ef5391d158c" containerName="payload-pre-processing"
Sep 09 22:09:21.785676 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785678    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a13563f7-de2c-4902-b8a5-5ef5391d158c" containerName="payload-pre-processing"
Sep 09 22:09:21.785779 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785691    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a45d201-a7a6-4e76-b6d0-79ac71a983b5" containerName="payload-processing"
Sep 09 22:09:21.785779 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785697    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a45d201-a7a6-4e76-b6d0-79ac71a983b5" containerName="payload-processing"
Sep 09 22:09:21.785779 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785703    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663c56f1-0bb0-48bc-a7fb-5daec275dc5a" containerName="revoke-keys"
Sep 09 22:09:21.785779 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785709    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="663c56f1-0bb0-48bc-a7fb-5daec275dc5a" containerName="revoke-keys"
Sep 09 22:09:21.785914 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785800    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a45d201-a7a6-4e76-b6d0-79ac71a983b5" containerName="payload-processing"
Sep 09 22:09:21.785914 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785807    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="663c56f1-0bb0-48bc-a7fb-5daec275dc5a" containerName="revoke-keys"
Sep 09 22:09:21.785914 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.785815    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a13563f7-de2c-4902-b8a5-5ef5391d158c" containerName="payload-pre-processing"
Sep 09 22:09:21.788595 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.788577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"
Sep 09 22:09:21.792106 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.792083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:09:21.801934 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.801911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"]
Sep 09 22:09:21.895657 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.895596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95344e1-13de-48a3-819f-536311002003-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz\" (UID: \"d95344e1-13de-48a3-819f-536311002003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"
Sep 09 22:09:21.996844 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.996810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95344e1-13de-48a3-819f-536311002003-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz\" (UID: \"d95344e1-13de-48a3-819f-536311002003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"
Sep 09 22:09:21.997435 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:21.997416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95344e1-13de-48a3-819f-536311002003-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz\" (UID: \"d95344e1-13de-48a3-819f-536311002003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"
Sep 09 22:09:22.099145 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:22.099061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"
Sep 09 22:09:22.374465 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:22.374389    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"]
Sep 09 22:09:22.378196 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:09:22.378165    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95344e1_13de_48a3_819f_536311002003.slice/crio-74859d594361642c1b36b8ff6c8e2864e58ce3bd4412d65dcceba0186d62060f WatchSource:0}: Error finding container 74859d594361642c1b36b8ff6c8e2864e58ce3bd4412d65dcceba0186d62060f: Status 404 returned error can't find the container with id 74859d594361642c1b36b8ff6c8e2864e58ce3bd4412d65dcceba0186d62060f
Sep 09 22:09:22.801495 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:22.801449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz" event={"ID":"d95344e1-13de-48a3-819f-536311002003","Type":"ContainerStarted","Data":"26bf0e4ecaab1b93f3a2ab99ab693df5a06a2989e5afe4efb2f9cae9bbfcf3e6"}
Sep 09 22:09:22.801983 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:22.801514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz" event={"ID":"d95344e1-13de-48a3-819f-536311002003","Type":"ContainerStarted","Data":"74859d594361642c1b36b8ff6c8e2864e58ce3bd4412d65dcceba0186d62060f"}
Sep 09 22:09:22.817241 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:22.817194    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz" podStartSLOduration=1.817179367 podStartE2EDuration="1.817179367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:09:22.816055559 +0000 UTC m=+1837.238418260" watchObservedRunningTime="2026-09-09 22:09:22.817179367 +0000 UTC m=+1837.239542066"
Sep 09 22:09:23.809474 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:23.809435    2579 generic.go:356] "Generic (PLEG): container finished" podID="d95344e1-13de-48a3-819f-536311002003" containerID="26bf0e4ecaab1b93f3a2ab99ab693df5a06a2989e5afe4efb2f9cae9bbfcf3e6" exitCode=0
Sep 09 22:09:23.809920 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:23.809531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz" event={"ID":"d95344e1-13de-48a3-819f-536311002003","Type":"ContainerDied","Data":"26bf0e4ecaab1b93f3a2ab99ab693df5a06a2989e5afe4efb2f9cae9bbfcf3e6"}
Sep 09 22:09:24.954094 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:24.954067    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"
Sep 09 22:09:25.027550 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:25.027514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95344e1-13de-48a3-819f-536311002003-maas-api-service-ca\") pod \"d95344e1-13de-48a3-819f-536311002003\" (UID: \"d95344e1-13de-48a3-819f-536311002003\") "
Sep 09 22:09:25.027935 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:25.027911    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95344e1-13de-48a3-819f-536311002003-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d95344e1-13de-48a3-819f-536311002003" (UID: "d95344e1-13de-48a3-819f-536311002003"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:25.129337 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:25.129302    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95344e1-13de-48a3-819f-536311002003-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:09:25.822982 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:25.822949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz" event={"ID":"d95344e1-13de-48a3-819f-536311002003","Type":"ContainerDied","Data":"74859d594361642c1b36b8ff6c8e2864e58ce3bd4412d65dcceba0186d62060f"}
Sep 09 22:09:25.822982 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:25.822976    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"
Sep 09 22:09:25.823226 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:25.822998    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74859d594361642c1b36b8ff6c8e2864e58ce3bd4412d65dcceba0186d62060f"
Sep 09 22:09:37.324044 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.324008    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"]
Sep 09 22:09:37.324553 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.324307    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb" podUID="6435b1f3-8fb2-42cc-a9f0-9fa7178403c9" containerName="payload-pre-processing" containerID="cri-o://51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc" gracePeriod=30
Sep 09 22:09:37.485816 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.485782    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"]
Sep 09 22:09:37.488325 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.488297    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2f2a52-99mqz"]
Sep 09 22:09:37.576645 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.576523    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:09:37.658564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.658532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwh98\" (UniqueName: \"kubernetes.io/projected/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-kube-api-access-hwh98\") pod \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\" (UID: \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\") "
Sep 09 22:09:37.658564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.658571    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-plugins-config-volume\") pod \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\" (UID: \"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9\") "
Sep 09 22:09:37.659007 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.658980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6435b1f3-8fb2-42cc-a9f0-9fa7178403c9" (UID: "6435b1f3-8fb2-42cc-a9f0-9fa7178403c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:09:37.660766 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.660744    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-kube-api-access-hwh98" (OuterVolumeSpecName: "kube-api-access-hwh98") pod "6435b1f3-8fb2-42cc-a9f0-9fa7178403c9" (UID: "6435b1f3-8fb2-42cc-a9f0-9fa7178403c9"). InnerVolumeSpecName "kube-api-access-hwh98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:09:37.760382 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.760338    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwh98\" (UniqueName: \"kubernetes.io/projected/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-kube-api-access-hwh98\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:09:37.760382 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.760368    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:09:37.900719 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.900594    2579 generic.go:356] "Generic (PLEG): container finished" podID="6435b1f3-8fb2-42cc-a9f0-9fa7178403c9" containerID="51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc" exitCode=0
Sep 09 22:09:37.900719 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.900719    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"
Sep 09 22:09:37.900962 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.900692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb" event={"ID":"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9","Type":"ContainerDied","Data":"51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc"}
Sep 09 22:09:37.900962 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.900857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb" event={"ID":"6435b1f3-8fb2-42cc-a9f0-9fa7178403c9","Type":"ContainerDied","Data":"5bf8e4520e085eadb6637b153019aeedc77687e9b79168c49c34e4edec5284ff"}
Sep 09 22:09:37.900962 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.900902    2579 scope.go:117] "RemoveContainer" containerID="51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc"
Sep 09 22:09:37.911128 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.911117    2579 scope.go:117] "RemoveContainer" containerID="51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc"
Sep 09 22:09:37.911394 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:09:37.911375    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc\": container with ID starting with 51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc not found: ID does not exist" containerID="51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc"
Sep 09 22:09:37.911440 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.911401    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc"} err="failed to get container status \"51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc\": rpc error: code = NotFound desc = could not find container \"51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc\": container with ID starting with 51a831f44d69ab28673c887c0fd36ccc6ffdf9321f557aeda49f0ddc5c723acc not found: ID does not exist"
Sep 09 22:09:37.925864 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.925841    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"]
Sep 09 22:09:37.929845 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:37.929816    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2f2a52-6ff574bbfd-bd4xb"]
Sep 09 22:09:38.092448 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:38.092413    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6435b1f3-8fb2-42cc-a9f0-9fa7178403c9" path="/var/lib/kubelet/pods/6435b1f3-8fb2-42cc-a9f0-9fa7178403c9/volumes"
Sep 09 22:09:38.092771 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:09:38.092756    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95344e1-13de-48a3-819f-536311002003" path="/var/lib/kubelet/pods/d95344e1-13de-48a3-819f-536311002003/volumes"
Sep 09 22:10:49.316150 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.316111    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"]
Sep 09 22:10:49.316745 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.316374    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy" containerID="cri-o://7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701" gracePeriod=30
Sep 09 22:10:49.706846 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.706805    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"]
Sep 09 22:10:49.707284 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.707271    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6435b1f3-8fb2-42cc-a9f0-9fa7178403c9" containerName="payload-pre-processing"
Sep 09 22:10:49.707330 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.707286    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6435b1f3-8fb2-42cc-a9f0-9fa7178403c9" containerName="payload-pre-processing"
Sep 09 22:10:49.707330 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.707310    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95344e1-13de-48a3-819f-536311002003" containerName="revoke-keys"
Sep 09 22:10:49.707330 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.707316    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95344e1-13de-48a3-819f-536311002003" containerName="revoke-keys"
Sep 09 22:10:49.707431 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.707387    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95344e1-13de-48a3-819f-536311002003" containerName="revoke-keys"
Sep 09 22:10:49.707431 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.707395    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6435b1f3-8fb2-42cc-a9f0-9fa7178403c9" containerName="payload-pre-processing"
Sep 09 22:10:49.710997 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.710977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"
Sep 09 22:10:49.713578 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.713555    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:10:49.718355 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.718323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"]
Sep 09 22:10:49.814758 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.814723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1734c64c-a7ab-40f5-bd60-6366123c5cbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h\" (UID: \"1734c64c-a7ab-40f5-bd60-6366123c5cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"
Sep 09 22:10:49.915692 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.915657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1734c64c-a7ab-40f5-bd60-6366123c5cbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h\" (UID: \"1734c64c-a7ab-40f5-bd60-6366123c5cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"
Sep 09 22:10:49.916260 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:49.916239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1734c64c-a7ab-40f5-bd60-6366123c5cbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h\" (UID: \"1734c64c-a7ab-40f5-bd60-6366123c5cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"
Sep 09 22:10:50.023226 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:50.023142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"
Sep 09 22:10:50.297265 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:50.297233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"]
Sep 09 22:10:50.298407 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:10:50.298377    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1734c64c_a7ab_40f5_bd60_6366123c5cbe.slice/crio-6862b517f2f986fd17f45f905397845a16e17193caa1b34503fe3660bfaaae57 WatchSource:0}: Error finding container 6862b517f2f986fd17f45f905397845a16e17193caa1b34503fe3660bfaaae57: Status 404 returned error can't find the container with id 6862b517f2f986fd17f45f905397845a16e17193caa1b34503fe3660bfaaae57
Sep 09 22:10:50.326349 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:50.326316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h" event={"ID":"1734c64c-a7ab-40f5-bd60-6366123c5cbe","Type":"ContainerStarted","Data":"6862b517f2f986fd17f45f905397845a16e17193caa1b34503fe3660bfaaae57"}
Sep 09 22:10:51.333515 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:51.333427    2579 generic.go:356] "Generic (PLEG): container finished" podID="1734c64c-a7ab-40f5-bd60-6366123c5cbe" containerID="52c45ac67616149078ede10d9679e54eadde6b1e301bdb8ec707c4c684da5118" exitCode=0
Sep 09 22:10:51.333515 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:51.333510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h" event={"ID":"1734c64c-a7ab-40f5-bd60-6366123c5cbe","Type":"ContainerDied","Data":"52c45ac67616149078ede10d9679e54eadde6b1e301bdb8ec707c4c684da5118"}
Sep 09 22:10:52.477897 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:52.477875    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"
Sep 09 22:10:52.540699 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:52.540663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1734c64c-a7ab-40f5-bd60-6366123c5cbe-maas-api-service-ca\") pod \"1734c64c-a7ab-40f5-bd60-6366123c5cbe\" (UID: \"1734c64c-a7ab-40f5-bd60-6366123c5cbe\") "
Sep 09 22:10:52.540991 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:52.540968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1734c64c-a7ab-40f5-bd60-6366123c5cbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1734c64c-a7ab-40f5-bd60-6366123c5cbe" (UID: "1734c64c-a7ab-40f5-bd60-6366123c5cbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:10:52.641549 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:52.641513    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1734c64c-a7ab-40f5-bd60-6366123c5cbe-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:53.346953 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:53.346910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h" event={"ID":"1734c64c-a7ab-40f5-bd60-6366123c5cbe","Type":"ContainerDied","Data":"6862b517f2f986fd17f45f905397845a16e17193caa1b34503fe3660bfaaae57"}
Sep 09 22:10:53.346953 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:53.346935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"
Sep 09 22:10:53.347181 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:53.346965    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6862b517f2f986fd17f45f905397845a16e17193caa1b34503fe3660bfaaae57"
Sep 09 22:10:54.580215 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.580150    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-data\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-certs\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662657    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-podinfo\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662691    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-credential-socket\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662735    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-token\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-954vq\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-kube-api-access-954vq\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451ec2a6-f3b9-4079-9dec-d92c10c89990-istiod-ca-cert\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-envoy\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.662917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-socket\") pod \"451ec2a6-f3b9-4079-9dec-d92c10c89990\" (UID: \"451ec2a6-f3b9-4079-9dec-d92c10c89990\") "
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.663796    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.664000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-data" (OuterVolumeSpecName: "istio-data") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.664201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.666854    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 22:10:54.667783 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.667072    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:10:54.670240 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.669382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-token" (OuterVolumeSpecName: "istio-token") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:10:54.670555 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.670522    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451ec2a6-f3b9-4079-9dec-d92c10c89990-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:10:54.671544 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.671505    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-kube-api-access-954vq" (OuterVolumeSpecName: "kube-api-access-954vq") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "kube-api-access-954vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:10:54.672190 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.672162    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "451ec2a6-f3b9-4079-9dec-d92c10c89990" (UID: "451ec2a6-f3b9-4079-9dec-d92c10c89990"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:10:54.764795 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764754    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:54.764795 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764790    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:54.764795 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764800    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:54.764795 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764807    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:54.765089 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764816    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:54.765089 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764825    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:54.765089 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764834    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-954vq\" (UniqueName: \"kubernetes.io/projected/451ec2a6-f3b9-4079-9dec-d92c10c89990-kube-api-access-954vq\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:54.765089 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764842    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451ec2a6-f3b9-4079-9dec-d92c10c89990-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:54.765089 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:54.764850    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451ec2a6-f3b9-4079-9dec-d92c10c89990-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:10:55.361495 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.361458    2579 generic.go:356] "Generic (PLEG): container finished" podID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerID="7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701" exitCode=0
Sep 09 22:10:55.361781 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.361516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" event={"ID":"451ec2a6-f3b9-4079-9dec-d92c10c89990","Type":"ContainerDied","Data":"7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701"}
Sep 09 22:10:55.361781 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.361522    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"
Sep 09 22:10:55.361781 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.361561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5" event={"ID":"451ec2a6-f3b9-4079-9dec-d92c10c89990","Type":"ContainerDied","Data":"6b0c8fc79fbad5d162197662850a7f365568b559071a481823bfa0ea1c924832"}
Sep 09 22:10:55.361781 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.361583    2579 scope.go:117] "RemoveContainer" containerID="7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701"
Sep 09 22:10:55.372637 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.372537    2579 scope.go:117] "RemoveContainer" containerID="a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480"
Sep 09 22:10:55.383239 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.382734    2579 scope.go:117] "RemoveContainer" containerID="7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701"
Sep 09 22:10:55.383547 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:10:55.383510    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701\": container with ID starting with 7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701 not found: ID does not exist" containerID="7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701"
Sep 09 22:10:55.383940 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.383553    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701"} err="failed to get container status \"7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701\": rpc error: code = NotFound desc = could not find container \"7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701\": container with ID starting with 7a84f7ff4a691724db57987137005242ffd11fc0fe1fb4d6d5d586c7f49da701 not found: ID does not exist"
Sep 09 22:10:55.383940 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.383572    2579 scope.go:117] "RemoveContainer" containerID="a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480"
Sep 09 22:10:55.384203 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:10:55.384173    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480\": container with ID starting with a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480 not found: ID does not exist" containerID="a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480"
Sep 09 22:10:55.384296 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.384212    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480"} err="failed to get container status \"a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480\": rpc error: code = NotFound desc = could not find container \"a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480\": container with ID starting with a4a8fd695610208d2fab18192453192b7c1c95036ac270744bca456beec84480 not found: ID does not exist"
Sep 09 22:10:55.386389 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.386367    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"]
Sep 09 22:10:55.388520 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:55.388501    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2f2a52-openshift-default-f4675455d-ncwz5"]
Sep 09 22:10:56.093619 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:10:56.093573    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" path="/var/lib/kubelet/pods/451ec2a6-f3b9-4079-9dec-d92c10c89990/volumes"
Sep 09 22:11:00.206573 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.206524    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"]
Sep 09 22:11:00.207158 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.206777    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp" podUID="d304d3a8-fb0f-451a-af89-1090dabf9136" containerName="payload-processing" containerID="cri-o://139a1584b854c8f06fac2a5b10ae4ebcab564af84c7c1f427f6f0968e416ff8a" gracePeriod=30
Sep 09 22:11:00.231517 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.231476    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"]
Sep 09 22:11:00.237471 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.231880    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l" podUID="c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da" containerName="payload-pre-processing" containerID="cri-o://064b8bbab3098e27e44f970903048e2b57756ef35d43d6ba8cde6a4e69476f2b" gracePeriod=30
Sep 09 22:11:00.363811 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.363778    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"]
Sep 09 22:11:00.367025 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.366996    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8fe6a1-xpw9h"]
Sep 09 22:11:00.395390 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.395353    2579 generic.go:356] "Generic (PLEG): container finished" podID="d304d3a8-fb0f-451a-af89-1090dabf9136" containerID="139a1584b854c8f06fac2a5b10ae4ebcab564af84c7c1f427f6f0968e416ff8a" exitCode=0
Sep 09 22:11:00.395552 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.395412    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp" event={"ID":"d304d3a8-fb0f-451a-af89-1090dabf9136","Type":"ContainerDied","Data":"139a1584b854c8f06fac2a5b10ae4ebcab564af84c7c1f427f6f0968e416ff8a"}
Sep 09 22:11:00.397413 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.397382    2579 generic.go:356] "Generic (PLEG): container finished" podID="c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da" containerID="064b8bbab3098e27e44f970903048e2b57756ef35d43d6ba8cde6a4e69476f2b" exitCode=0
Sep 09 22:11:00.397413 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.397408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l" event={"ID":"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da","Type":"ContainerDied","Data":"064b8bbab3098e27e44f970903048e2b57756ef35d43d6ba8cde6a4e69476f2b"}
Sep 09 22:11:00.490664 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.490590    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:11:00.522245 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.522213    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:11:00.526594 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.526533    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq5kd\" (UniqueName: \"kubernetes.io/projected/d304d3a8-fb0f-451a-af89-1090dabf9136-kube-api-access-qq5kd\") pod \"d304d3a8-fb0f-451a-af89-1090dabf9136\" (UID: \"d304d3a8-fb0f-451a-af89-1090dabf9136\") "
Sep 09 22:11:00.526777 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.526647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d304d3a8-fb0f-451a-af89-1090dabf9136-plugins-config-volume\") pod \"d304d3a8-fb0f-451a-af89-1090dabf9136\" (UID: \"d304d3a8-fb0f-451a-af89-1090dabf9136\") "
Sep 09 22:11:00.527202 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.527171    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d304d3a8-fb0f-451a-af89-1090dabf9136-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d304d3a8-fb0f-451a-af89-1090dabf9136" (UID: "d304d3a8-fb0f-451a-af89-1090dabf9136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:00.529089 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.529065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d304d3a8-fb0f-451a-af89-1090dabf9136-kube-api-access-qq5kd" (OuterVolumeSpecName: "kube-api-access-qq5kd") pod "d304d3a8-fb0f-451a-af89-1090dabf9136" (UID: "d304d3a8-fb0f-451a-af89-1090dabf9136"). InnerVolumeSpecName "kube-api-access-qq5kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:00.627851 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.627819    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-plugins-config-volume\") pod \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\" (UID: \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\") "
Sep 09 22:11:00.628029 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.627870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tlqd\" (UniqueName: \"kubernetes.io/projected/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-kube-api-access-2tlqd\") pod \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\" (UID: \"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da\") "
Sep 09 22:11:00.628110 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.628098    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5kd\" (UniqueName: \"kubernetes.io/projected/d304d3a8-fb0f-451a-af89-1090dabf9136-kube-api-access-qq5kd\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:00.628149 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.628111    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d304d3a8-fb0f-451a-af89-1090dabf9136-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:00.628185 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.628150    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da" (UID: "c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:00.629853 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.629833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-kube-api-access-2tlqd" (OuterVolumeSpecName: "kube-api-access-2tlqd") pod "c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da" (UID: "c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da"). InnerVolumeSpecName "kube-api-access-2tlqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:00.729696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.729559    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:00.729696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:00.729596    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlqd\" (UniqueName: \"kubernetes.io/projected/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da-kube-api-access-2tlqd\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:01.404556 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.404524    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"
Sep 09 22:11:01.404556 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.404530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp" event={"ID":"d304d3a8-fb0f-451a-af89-1090dabf9136","Type":"ContainerDied","Data":"310df171ae3f3e8e3e54e9f7301713afa33c14ba995e16a94a53355c36f43af6"}
Sep 09 22:11:01.405096 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.404595    2579 scope.go:117] "RemoveContainer" containerID="139a1584b854c8f06fac2a5b10ae4ebcab564af84c7c1f427f6f0968e416ff8a"
Sep 09 22:11:01.406058 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.405930    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"
Sep 09 22:11:01.406058 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.405978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l" event={"ID":"c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da","Type":"ContainerDied","Data":"fdbddd76ab460516bf6b3857b6ef5925c24336df6f9f7e984eaf21339da41a0d"}
Sep 09 22:11:01.415806 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.415784    2579 scope.go:117] "RemoveContainer" containerID="064b8bbab3098e27e44f970903048e2b57756ef35d43d6ba8cde6a4e69476f2b"
Sep 09 22:11:01.429293 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.429270    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"]
Sep 09 22:11:01.433702 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.433680    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8fe6a1-6f77b57997-gsfgp"]
Sep 09 22:11:01.445031 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.445006    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"]
Sep 09 22:11:01.454052 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:01.454030    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8fe6a1-d67cd64b6-fzl8l"]
Sep 09 22:11:02.092732 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:02.092691    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1734c64c-a7ab-40f5-bd60-6366123c5cbe" path="/var/lib/kubelet/pods/1734c64c-a7ab-40f5-bd60-6366123c5cbe/volumes"
Sep 09 22:11:02.093209 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:02.093187    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da" path="/var/lib/kubelet/pods/c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da/volumes"
Sep 09 22:11:02.093612 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:02.093579    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d304d3a8-fb0f-451a-af89-1090dabf9136" path="/var/lib/kubelet/pods/d304d3a8-fb0f-451a-af89-1090dabf9136/volumes"
Sep 09 22:11:06.963228 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963194    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"]
Sep 09 22:11:06.963696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963648    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1734c64c-a7ab-40f5-bd60-6366123c5cbe" containerName="revoke-keys"
Sep 09 22:11:06.963696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963662    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1734c64c-a7ab-40f5-bd60-6366123c5cbe" containerName="revoke-keys"
Sep 09 22:11:06.963696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963673    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da" containerName="payload-pre-processing"
Sep 09 22:11:06.963696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963678    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da" containerName="payload-pre-processing"
Sep 09 22:11:06.963696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963691    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d304d3a8-fb0f-451a-af89-1090dabf9136" containerName="payload-processing"
Sep 09 22:11:06.963696 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963696    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d304d3a8-fb0f-451a-af89-1090dabf9136" containerName="payload-processing"
Sep 09 22:11:06.963908 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963708    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy"
Sep 09 22:11:06.963908 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963714    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy"
Sep 09 22:11:06.963908 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963784    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1734c64c-a7ab-40f5-bd60-6366123c5cbe" containerName="revoke-keys"
Sep 09 22:11:06.963908 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963792    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d304d3a8-fb0f-451a-af89-1090dabf9136" containerName="payload-processing"
Sep 09 22:11:06.963908 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963797    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy"
Sep 09 22:11:06.963908 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963803    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy"
Sep 09 22:11:06.963908 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.963809    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c16dbbdb-21a0-4ed3-a0c4-3dd96d52d2da" containerName="payload-pre-processing"
Sep 09 22:11:06.968409 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.968392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"
Sep 09 22:11:06.972078 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.972054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqcf2\""
Sep 09 22:11:06.974250 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:06.974226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"]
Sep 09 22:11:07.094050 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:07.094012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3d61-26fa-48c5-a045-305da26d802a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt\" (UID: \"df9d3d61-26fa-48c5-a045-305da26d802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"
Sep 09 22:11:07.195660 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:07.195596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3d61-26fa-48c5-a045-305da26d802a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt\" (UID: \"df9d3d61-26fa-48c5-a045-305da26d802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"
Sep 09 22:11:07.196733 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:07.196483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3d61-26fa-48c5-a045-305da26d802a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt\" (UID: \"df9d3d61-26fa-48c5-a045-305da26d802a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"
Sep 09 22:11:07.280678 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:07.280555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"
Sep 09 22:11:07.554064 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:07.554036    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"]
Sep 09 22:11:07.555691 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:11:07.555657    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9d3d61_26fa_48c5_a045_305da26d802a.slice/crio-9b2b19824b4d745f08897f85a0bc87c4b96a6617fd3baac4b8c13f6a16c62d12 WatchSource:0}: Error finding container 9b2b19824b4d745f08897f85a0bc87c4b96a6617fd3baac4b8c13f6a16c62d12: Status 404 returned error can't find the container with id 9b2b19824b4d745f08897f85a0bc87c4b96a6617fd3baac4b8c13f6a16c62d12
Sep 09 22:11:08.448915 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:08.448828    2579 generic.go:356] "Generic (PLEG): container finished" podID="df9d3d61-26fa-48c5-a045-305da26d802a" containerID="c976ef505fcace21b20d978961ac035757ba3f2fbcf88dd4dc8ba9f987e47cfa" exitCode=0
Sep 09 22:11:08.449288 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:08.448923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt" event={"ID":"df9d3d61-26fa-48c5-a045-305da26d802a","Type":"ContainerDied","Data":"c976ef505fcace21b20d978961ac035757ba3f2fbcf88dd4dc8ba9f987e47cfa"}
Sep 09 22:11:08.449288 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:08.448986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt" event={"ID":"df9d3d61-26fa-48c5-a045-305da26d802a","Type":"ContainerStarted","Data":"9b2b19824b4d745f08897f85a0bc87c4b96a6617fd3baac4b8c13f6a16c62d12"}
Sep 09 22:11:09.592574 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:09.592549    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"
Sep 09 22:11:09.719668 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:09.719569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3d61-26fa-48c5-a045-305da26d802a-maas-api-service-ca\") pod \"df9d3d61-26fa-48c5-a045-305da26d802a\" (UID: \"df9d3d61-26fa-48c5-a045-305da26d802a\") "
Sep 09 22:11:09.719907 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:09.719882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9d3d61-26fa-48c5-a045-305da26d802a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df9d3d61-26fa-48c5-a045-305da26d802a" (UID: "df9d3d61-26fa-48c5-a045-305da26d802a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:09.820596 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:09.820552    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9d3d61-26fa-48c5-a045-305da26d802a-maas-api-service-ca\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:10.464487 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:10.464390    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt" event={"ID":"df9d3d61-26fa-48c5-a045-305da26d802a","Type":"ContainerDied","Data":"9b2b19824b4d745f08897f85a0bc87c4b96a6617fd3baac4b8c13f6a16c62d12"}
Sep 09 22:11:10.464487 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:10.464445    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b2b19824b4d745f08897f85a0bc87c4b96a6617fd3baac4b8c13f6a16c62d12"
Sep 09 22:11:10.464487 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:10.464409    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"
Sep 09 22:11:17.457937 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.457897    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"]
Sep 09 22:11:17.462756 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.462725    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ace629-29816520-4d9zr"]
Sep 09 22:11:17.490268 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.490231    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"]
Sep 09 22:11:17.490567 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.490537    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6" podUID="9809528d-8b1c-4e09-86e7-d1fef6d9f559" containerName="payload-pre-processing" containerID="cri-o://6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc" gracePeriod=30
Sep 09 22:11:17.653319 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.653287    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"]
Sep 09 22:11:17.655328 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.655302    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ace629-glnzt"]
Sep 09 22:11:17.753561 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.753535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 22:11:17.907353 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.907305    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vj79\" (UniqueName: \"kubernetes.io/projected/9809528d-8b1c-4e09-86e7-d1fef6d9f559-kube-api-access-6vj79\") pod \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\" (UID: \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\") "
Sep 09 22:11:17.907552 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.907394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809528d-8b1c-4e09-86e7-d1fef6d9f559-plugins-config-volume\") pod \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\" (UID: \"9809528d-8b1c-4e09-86e7-d1fef6d9f559\") "
Sep 09 22:11:17.907824 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.907800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9809528d-8b1c-4e09-86e7-d1fef6d9f559-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9809528d-8b1c-4e09-86e7-d1fef6d9f559" (UID: "9809528d-8b1c-4e09-86e7-d1fef6d9f559"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:17.909565 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:17.909542    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9809528d-8b1c-4e09-86e7-d1fef6d9f559-kube-api-access-6vj79" (OuterVolumeSpecName: "kube-api-access-6vj79") pod "9809528d-8b1c-4e09-86e7-d1fef6d9f559" (UID: "9809528d-8b1c-4e09-86e7-d1fef6d9f559"). InnerVolumeSpecName "kube-api-access-6vj79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:18.008528 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.008438    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vj79\" (UniqueName: \"kubernetes.io/projected/9809528d-8b1c-4e09-86e7-d1fef6d9f559-kube-api-access-6vj79\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:18.008528 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.008470    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809528d-8b1c-4e09-86e7-d1fef6d9f559-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:18.092441 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.092404    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e218407-dfa7-41bb-918f-e1e56f787687" path="/var/lib/kubelet/pods/4e218407-dfa7-41bb-918f-e1e56f787687/volumes"
Sep 09 22:11:18.092779 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.092763    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9d3d61-26fa-48c5-a045-305da26d802a" path="/var/lib/kubelet/pods/df9d3d61-26fa-48c5-a045-305da26d802a/volumes"
Sep 09 22:11:18.514904 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.514861    2579 generic.go:356] "Generic (PLEG): container finished" podID="9809528d-8b1c-4e09-86e7-d1fef6d9f559" containerID="6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc" exitCode=0
Sep 09 22:11:18.514904 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.514909    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"
Sep 09 22:11:18.515476 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.514951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6" event={"ID":"9809528d-8b1c-4e09-86e7-d1fef6d9f559","Type":"ContainerDied","Data":"6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc"}
Sep 09 22:11:18.515476 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.515019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6" event={"ID":"9809528d-8b1c-4e09-86e7-d1fef6d9f559","Type":"ContainerDied","Data":"b1ea6a38328d1917fbf594813324267bef6786478c75b27a1ba06b774c4bbd92"}
Sep 09 22:11:18.515476 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.515054    2579 scope.go:117] "RemoveContainer" containerID="6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc"
Sep 09 22:11:18.524393 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.524371    2579 scope.go:117] "RemoveContainer" containerID="6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc"
Sep 09 22:11:18.524731 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:11:18.524707    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc\": container with ID starting with 6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc not found: ID does not exist" containerID="6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc"
Sep 09 22:11:18.524800 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.524742    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc"} err="failed to get container status \"6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc\": rpc error: code = NotFound desc = could not find container \"6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc\": container with ID starting with 6eb5d7bfe161e4ec6126bb29e45f22e4851e1cb1aebae19a2f446fd07d0543cc not found: ID does not exist"
Sep 09 22:11:18.537789 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.537760    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"]
Sep 09 22:11:18.542614 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:18.542578    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ace629-86b46d557d-56pv6"]
Sep 09 22:11:20.092556 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:20.092518    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9809528d-8b1c-4e09-86e7-d1fef6d9f559" path="/var/lib/kubelet/pods/9809528d-8b1c-4e09-86e7-d1fef6d9f559/volumes"
Sep 09 22:11:24.152760 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:24.152728    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"]
Sep 09 22:11:24.153215 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:24.152931    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" containerID="cri-o://02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299" gracePeriod=30
Sep 09 22:11:29.414426 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.414394    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 22:11:29.517958 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.517867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhgkx\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-kube-api-access-vhgkx\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.517958 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.517918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-envoy\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.517958 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.517945    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-credential-socket\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.518225 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.517970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istiod-ca-cert\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.518225 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-data\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.518225 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518081    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-certs\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.518225 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518111    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-podinfo\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.518225 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-token\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.518225 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-socket\") pod \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\" (UID: \"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7\") "
Sep 09 22:11:29.518535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:29.518535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518351    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:29.518535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518429    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:29.518535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:29.518788 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518572    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:29.518788 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518690    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:29.518788 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518707    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:29.518788 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518722    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:29.518788 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518736    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:29.518788 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.518752    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:29.520237 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.520206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-kube-api-access-vhgkx" (OuterVolumeSpecName: "kube-api-access-vhgkx") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "kube-api-access-vhgkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:29.520397 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.520376    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:29.520490 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.520471    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:29.520531 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.520516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" (UID: "3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 22:11:29.586534 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.586494    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerID="02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299" exitCode=0
Sep 09 22:11:29.586763 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.586551    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"
Sep 09 22:11:29.586763 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.586591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" event={"ID":"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7","Type":"ContainerDied","Data":"02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299"}
Sep 09 22:11:29.586763 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.586677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" event={"ID":"3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7","Type":"ContainerDied","Data":"d2bae3c994686f0b52cca6f77a03d2a5064f876e0d210e3d1426f6a8dc7d3595"}
Sep 09 22:11:29.586763 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.586708    2579 scope.go:117] "RemoveContainer" containerID="02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299"
Sep 09 22:11:29.596966 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.596945    2579 scope.go:117] "RemoveContainer" containerID="be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa"
Sep 09 22:11:29.605922 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.605901    2579 scope.go:117] "RemoveContainer" containerID="02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299"
Sep 09 22:11:29.606237 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:11:29.606217    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299\": container with ID starting with 02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299 not found: ID does not exist" containerID="02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299"
Sep 09 22:11:29.606291 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.606247    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299"} err="failed to get container status \"02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299\": rpc error: code = NotFound desc = could not find container \"02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299\": container with ID starting with 02aaa1930774476de50a6a9e7ee5aabe74dfdd8ca06b00676b46d1a7433bf299 not found: ID does not exist"
Sep 09 22:11:29.606291 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.606266    2579 scope.go:117] "RemoveContainer" containerID="be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa"
Sep 09 22:11:29.606579 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:11:29.606558    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa\": container with ID starting with be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa not found: ID does not exist" containerID="be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa"
Sep 09 22:11:29.606709 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.606678    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa"} err="failed to get container status \"be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa\": rpc error: code = NotFound desc = could not find container \"be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa\": container with ID starting with be0f5b12d61b67458665ebd0c94c007758784c98a75489405bcd1168c90987fa not found: ID does not exist"
Sep 09 22:11:29.609563 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.609538    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"]
Sep 09 22:11:29.615506 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.615479    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4"]
Sep 09 22:11:29.619575 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.619556    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:29.619643 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.619578    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:29.619643 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.619588    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhgkx\" (UniqueName: \"kubernetes.io/projected/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-kube-api-access-vhgkx\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:29.619643 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:29.619598    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:30.092438 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:30.092392    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" path="/var/lib/kubelet/pods/3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7/volumes"
Sep 09 22:11:30.405290 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:30.405244    2579 patch_prober.go:28] interesting pod/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 22:11:30.405473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:30.405314    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ace629-openshift-default-659f87c768-zfkk4" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 22:11:34.968204 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:34.968154    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"]
Sep 09 22:11:34.970765 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:34.970727    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3b2190-29816520-jjgn5"]
Sep 09 22:11:34.994146 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:34.994110    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"]
Sep 09 22:11:34.994381 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:34.994336    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq" podUID="a24bf8b2-a02f-4148-b098-2b820fed9147" containerName="payload-processing" containerID="cri-o://84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c" gracePeriod=30
Sep 09 22:11:35.261793 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.261767    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 22:11:35.377968 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.377934    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24bf8b2-a02f-4148-b098-2b820fed9147-plugins-config-volume\") pod \"a24bf8b2-a02f-4148-b098-2b820fed9147\" (UID: \"a24bf8b2-a02f-4148-b098-2b820fed9147\") "
Sep 09 22:11:35.378164 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.377983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w47gj\" (UniqueName: \"kubernetes.io/projected/a24bf8b2-a02f-4148-b098-2b820fed9147-kube-api-access-w47gj\") pod \"a24bf8b2-a02f-4148-b098-2b820fed9147\" (UID: \"a24bf8b2-a02f-4148-b098-2b820fed9147\") "
Sep 09 22:11:35.378368 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.378346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24bf8b2-a02f-4148-b098-2b820fed9147-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a24bf8b2-a02f-4148-b098-2b820fed9147" (UID: "a24bf8b2-a02f-4148-b098-2b820fed9147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:35.380219 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.380195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24bf8b2-a02f-4148-b098-2b820fed9147-kube-api-access-w47gj" (OuterVolumeSpecName: "kube-api-access-w47gj") pod "a24bf8b2-a02f-4148-b098-2b820fed9147" (UID: "a24bf8b2-a02f-4148-b098-2b820fed9147"). InnerVolumeSpecName "kube-api-access-w47gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:35.479782 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.479745    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24bf8b2-a02f-4148-b098-2b820fed9147-plugins-config-volume\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:35.479782 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.479771    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w47gj\" (UniqueName: \"kubernetes.io/projected/a24bf8b2-a02f-4148-b098-2b820fed9147-kube-api-access-w47gj\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:35.623593 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.623553    2579 generic.go:356] "Generic (PLEG): container finished" podID="a24bf8b2-a02f-4148-b098-2b820fed9147" containerID="84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c" exitCode=0
Sep 09 22:11:35.623767 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.623618    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"
Sep 09 22:11:35.623767 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.623641    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq" event={"ID":"a24bf8b2-a02f-4148-b098-2b820fed9147","Type":"ContainerDied","Data":"84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c"}
Sep 09 22:11:35.623767 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.623700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq" event={"ID":"a24bf8b2-a02f-4148-b098-2b820fed9147","Type":"ContainerDied","Data":"77fe046ac5dba8dccb07bcac5e765e06001160fb667f8bacef1cf95c15750697"}
Sep 09 22:11:35.623767 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.623717    2579 scope.go:117] "RemoveContainer" containerID="84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c"
Sep 09 22:11:35.633460 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.633438    2579 scope.go:117] "RemoveContainer" containerID="84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c"
Sep 09 22:11:35.633764 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:11:35.633741    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c\": container with ID starting with 84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c not found: ID does not exist" containerID="84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c"
Sep 09 22:11:35.633852 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.633774    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c"} err="failed to get container status \"84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c\": rpc error: code = NotFound desc = could not find container \"84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c\": container with ID starting with 84d0697f439d43bea2b19f221635a9a223d8a604b9bcaa4b986c308903a9ae4c not found: ID does not exist"
Sep 09 22:11:35.645781 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.645753    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"]
Sep 09 22:11:35.651998 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:35.651969    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b2190-85c6654f55-ks5lq"]
Sep 09 22:11:36.092657 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:36.092568    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a20807a8-ea12-4368-a3f7-571ba484b516" path="/var/lib/kubelet/pods/a20807a8-ea12-4368-a3f7-571ba484b516/volumes"
Sep 09 22:11:36.093121 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:36.092916    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24bf8b2-a02f-4148-b098-2b820fed9147" path="/var/lib/kubelet/pods/a24bf8b2-a02f-4148-b098-2b820fed9147/volumes"
Sep 09 22:11:41.905537 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:41.905447    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"]
Sep 09 22:11:41.906013 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:41.905737    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy" containerID="cri-o://6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9" gracePeriod=30
Sep 09 22:11:43.897886 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:43.897840    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 22:11:43.898280 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:43.897909    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 22:11:46.497117 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:46.497086    2579 scope.go:117] "RemoveContainer" containerID="cff57fb3c6aedb33390a6466b40fc977593cbc4ea628c70a8f59fde07be066ab"
Sep 09 22:11:46.507145 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:46.507122    2579 scope.go:117] "RemoveContainer" containerID="1b5c9c93320e3ec0ecce273d48027dbdfcf4f528448515ed89d9987612c43979"
Sep 09 22:11:47.161204 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.161171    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 22:11:47.297260 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297152    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-data\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297260 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297221    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d29cb97-241e-492f-92fb-d7702847c2f6-istiod-ca-cert\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297260 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297252    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-socket\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297300    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-certs\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297370    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-podinfo\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-credential-socket\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-token\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4gqs\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-kube-api-access-j4gqs\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297492    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:47.297564 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-envoy\") pod \"2d29cb97-241e-492f-92fb-d7702847c2f6\" (UID: \"2d29cb97-241e-492f-92fb-d7702847c2f6\") "
Sep 09 22:11:47.297951 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297576    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:47.297951 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297643    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:47.297951 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297744    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d29cb97-241e-492f-92fb-d7702847c2f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:11:47.297951 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:47.298176 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297953    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d29cb97-241e-492f-92fb-d7702847c2f6-istiod-ca-cert\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.298176 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297972    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.298176 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.297987    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-workload-certs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.298176 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.298001    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-credential-socket\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.298176 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.298013    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-data\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.299974 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.299942    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-kube-api-access-j4gqs" (OuterVolumeSpecName: "kube-api-access-j4gqs") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "kube-api-access-j4gqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:47.300121 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.299974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:11:47.300257 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.300227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:11:47.300367 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.300307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d29cb97-241e-492f-92fb-d7702847c2f6" (UID: "2d29cb97-241e-492f-92fb-d7702847c2f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 22:11:47.399218 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.399168    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-podinfo\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.399218 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.399216    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-token\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.399441 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.399230    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gqs\" (UniqueName: \"kubernetes.io/projected/2d29cb97-241e-492f-92fb-d7702847c2f6-kube-api-access-j4gqs\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.399441 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.399245    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d29cb97-241e-492f-92fb-d7702847c2f6-istio-envoy\") on node \"ip-10-0-139-221.ec2.internal\" DevicePath \"\""
Sep 09 22:11:47.690998 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.690960    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerID="6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9" exitCode=0
Sep 09 22:11:47.691402 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.691016    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"
Sep 09 22:11:47.691402 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.691053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" event={"ID":"2d29cb97-241e-492f-92fb-d7702847c2f6","Type":"ContainerDied","Data":"6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9"}
Sep 09 22:11:47.691402 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.691117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd" event={"ID":"2d29cb97-241e-492f-92fb-d7702847c2f6","Type":"ContainerDied","Data":"4a992b0460e86b342f041d9960aa32fb43657602a4c738b1c6aa658c7b80dd28"}
Sep 09 22:11:47.691402 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.691147    2579 scope.go:117] "RemoveContainer" containerID="6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9"
Sep 09 22:11:47.701074 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.701053    2579 scope.go:117] "RemoveContainer" containerID="7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d"
Sep 09 22:11:47.709911 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.709887    2579 scope.go:117] "RemoveContainer" containerID="6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9"
Sep 09 22:11:47.710195 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:11:47.710174    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9\": container with ID starting with 6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9 not found: ID does not exist" containerID="6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9"
Sep 09 22:11:47.710284 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.710207    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9"} err="failed to get container status \"6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9\": rpc error: code = NotFound desc = could not find container \"6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9\": container with ID starting with 6266d86bd31a8f2855458ad9d6fb66ff0ffe2323bb7c6237f532a56f7ba333c9 not found: ID does not exist"
Sep 09 22:11:47.710284 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.710231    2579 scope.go:117] "RemoveContainer" containerID="7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d"
Sep 09 22:11:47.710509 ip-10-0-139-221 kubenswrapper[2579]: E0909 22:11:47.710488    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d\": container with ID starting with 7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d not found: ID does not exist" containerID="7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d"
Sep 09 22:11:47.710549 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.710515    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d"} err="failed to get container status \"7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d\": rpc error: code = NotFound desc = could not find container \"7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d\": container with ID starting with 7cd4023f1257b6d18754bb800402196372bcc741df73ee2e026c1818ad5ea52d not found: ID does not exist"
Sep 09 22:11:47.714738 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.714713    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"]
Sep 09 22:11:47.719541 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:47.719517    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b2190-openshift-default-577b54986d-qvhfd"]
Sep 09 22:11:48.093006 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:11:48.092927    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" path="/var/lib/kubelet/pods/2d29cb97-241e-492f-92fb-d7702847c2f6/volumes"
Sep 09 22:12:46.561304 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:12:46.561271    2579 scope.go:117] "RemoveContainer" containerID="67a3dc334e598c503342a54c4fce8d72764acde1151c04832a4d84c9bc2cd056"
Sep 09 22:12:46.569916 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:12:46.569893    2579 scope.go:117] "RemoveContainer" containerID="5372062d0b5102eb7853873da52f474f8d19458e5f965d2365e31a6589fb135c"
Sep 09 22:13:46.597214 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:13:46.597179    2579 scope.go:117] "RemoveContainer" containerID="f53d0f5e61d83b9cb4664af15a260b563e79972f1503f69388ac506825a99075"
Sep 09 22:13:46.605943 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:13:46.605925    2579 scope.go:117] "RemoveContainer" containerID="f04641200e40d5d05fe38cf7f3fa1b9e1dbd6cc7d374da4177ffb90c94eed5aa"
Sep 09 22:15:46.637261 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:15:46.637230    2579 scope.go:117] "RemoveContainer" containerID="b53e203b7a44652efad34ab837356aded2d51233d94eed5c602ef28373d600d1"
Sep 09 22:15:46.645687 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:15:46.645669    2579 scope.go:117] "RemoveContainer" containerID="26bf0e4ecaab1b93f3a2ab99ab693df5a06a2989e5afe4efb2f9cae9bbfcf3e6"
Sep 09 22:17:20.690624 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.690569    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"]
Sep 09 22:17:20.691292 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691239    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy"
Sep 09 22:17:20.691292 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691265    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy"
Sep 09 22:17:20.691292 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691282    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy"
Sep 09 22:17:20.691292 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691291    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691310    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24bf8b2-a02f-4148-b098-2b820fed9147" containerName="payload-processing"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691319    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24bf8b2-a02f-4148-b098-2b820fed9147" containerName="payload-processing"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691330    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691339    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="451ec2a6-f3b9-4079-9dec-d92c10c89990" containerName="istio-proxy"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691348    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9d3d61-26fa-48c5-a045-305da26d802a" containerName="revoke-keys"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691356    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9d3d61-26fa-48c5-a045-305da26d802a" containerName="revoke-keys"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691371    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9809528d-8b1c-4e09-86e7-d1fef6d9f559" containerName="payload-pre-processing"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691379    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9809528d-8b1c-4e09-86e7-d1fef6d9f559" containerName="payload-pre-processing"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691391    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691399    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691514    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9809528d-8b1c-4e09-86e7-d1fef6d9f559" containerName="payload-pre-processing"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691527    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9d3d61-26fa-48c5-a045-305da26d802a" containerName="revoke-keys"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691537    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691547    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24bf8b2-a02f-4148-b098-2b820fed9147" containerName="payload-processing"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691557    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed05a1f-0e85-48ec-8c0d-dbc9159e86b7" containerName="istio-proxy"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691571    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy"
Sep 09 22:17:20.691586 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.691581    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy"
Sep 09 22:17:20.697381 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.697359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.700970 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.700949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-94w96\""
Sep 09 22:17:20.701095 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.700953    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 22:17:20.701095 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.700955    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 22:17:20.705441 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.705415    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"]
Sep 09 22:17:20.848227 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.848194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-metrics-certs\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.848413 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.848234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-maas-api-tls\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.848413 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.848266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-openshift-service-ca\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.848413 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.848325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sp9p\" (UniqueName: \"kubernetes.io/projected/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-kube-api-access-2sp9p\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.949081 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.948990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-metrics-certs\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.949081 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.949036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-maas-api-tls\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.949081 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.949076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-openshift-service-ca\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.949377 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.949111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sp9p\" (UniqueName: \"kubernetes.io/projected/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-kube-api-access-2sp9p\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.949932 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.949903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-openshift-service-ca\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.951362 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.951340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-metrics-certs\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.951462 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.951390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-maas-api-tls\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:20.956511 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:20.956489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sp9p\" (UniqueName: \"kubernetes.io/projected/2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81-kube-api-access-2sp9p\") pod \"maas-api-77fc54c768-kjf5f\" (UID: \"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81\") " pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:21.010836 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:21.010803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:21.283199 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:21.283171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"]
Sep 09 22:17:21.285259 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:17:21.285226    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea3c5e9_8c5f_4f2c_9245_c79b07ba4a81.slice/crio-50c5fd9eab077f38e0a5745363055fd92f64fc05bf05b2775d485b273b26381d WatchSource:0}: Error finding container 50c5fd9eab077f38e0a5745363055fd92f64fc05bf05b2775d485b273b26381d: Status 404 returned error can't find the container with id 50c5fd9eab077f38e0a5745363055fd92f64fc05bf05b2775d485b273b26381d
Sep 09 22:17:21.286652 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:21.286631    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 22:17:21.425250 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:21.425217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f" event={"ID":"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81","Type":"ContainerStarted","Data":"50c5fd9eab077f38e0a5745363055fd92f64fc05bf05b2775d485b273b26381d"}
Sep 09 22:17:24.445733 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:24.445693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f" event={"ID":"2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81","Type":"ContainerStarted","Data":"92562a195a129e6ba39b9d52158119d6786a74e011836f14668dd90b8e711915"}
Sep 09 22:17:24.446136 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:24.445762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:24.464560 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:24.464510    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f" podStartSLOduration=2.077745656 podStartE2EDuration="4.464491894s" totalImagesPullingTime="2.386746238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:17:24.462817579 +0000 UTC m=+2318.885180315" watchObservedRunningTime="2026-09-09 22:17:24.464491894 +0000 UTC m=+2318.886854650"
Sep 09 22:17:30.458936 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:30.458906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77fc54c768-kjf5f"
Sep 09 22:17:46.674542 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:46.674465    2579 scope.go:117] "RemoveContainer" containerID="52c45ac67616149078ede10d9679e54eadde6b1e301bdb8ec707c4c684da5118"
Sep 09 22:17:46.682769 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:46.682751    2579 scope.go:117] "RemoveContainer" containerID="c976ef505fcace21b20d978961ac035757ba3f2fbcf88dd4dc8ba9f987e47cfa"
Sep 09 22:17:55.489582 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:55.489546    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ndh64_bb9f6ae7-1d80-4c29-899f-0f935069a4ab/manager/0.log"
Sep 09 22:17:55.597382 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:55.597350    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jmpp8_478bfb26-f142-4867-a37e-089756d3809b/manager/0.log"
Sep 09 22:17:56.180342 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:56.180312    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56bc947947-gwfqd_81f4008d-743b-4c71-a8b6-3c24545ab2a8/copy-manifests/0.log"
Sep 09 22:17:56.186192 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:56.186165    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56bc947947-gwfqd_81f4008d-743b-4c71-a8b6-3c24545ab2a8/manager/0.log"
Sep 09 22:17:56.779588 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:56.779554    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77fc54c768-kjf5f_2ea3c5e9-8c5f-4f2c-9245-c79b07ba4a81/maas-api/0.log"
Sep 09 22:17:56.900742 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:56.900714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816520-lbsmq_e530b8fb-835e-4edb-976f-87debade942f/cleanup/0.log"
Sep 09 22:17:57.147073 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:57.147041    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kv6k9_fc5257cb-82c7-48c1-9937-be1be1e2a18c/postgres/0.log"
Sep 09 22:17:58.264622 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:58.264584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pnqnf_7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2/kuadrant-console-plugin/0.log"
Sep 09 22:17:58.608046 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:58.608017    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-sp7fz_7167994c-bafe-4e71-81df-570f550e40a4/manager/0.log"
Sep 09 22:17:58.955585 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:58.955504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9_bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4/istio-proxy/0.log"
Sep 09 22:17:59.417010 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:59.416972    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7vd78_df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc/istio-proxy/1.log"
Sep 09 22:17:59.538254 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:59.538221    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6jbpm_6e710c3b-e23c-4789-ba15-0088d6d84359/payload-pre-processing/0.log"
Sep 09 22:17:59.776204 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:17:59.776088    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c55bbf95b-wdt7z_5613aec3-899d-4cbf-a406-7e1d4b0f5bd7/router/0.log"
Sep 09 22:18:00.318277 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:00.318248    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-8j5lv_f7d7b5f1-7797-4122-8d11-2021c1d3db3e/main/0.log"
Sep 09 22:18:00.432531 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:00.432487    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4994x_b250c38e-6f70-464b-9d57-2289baf2d3e9/main/0.log"
Sep 09 22:18:08.826332 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:08.826296    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9kpfp_99c997d6-cc19-4559-8591-58462280463f/global-pull-secret-syncer/0.log"
Sep 09 22:18:09.026452 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:09.026409    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fpkj8_fb8970da-0085-406e-b96f-8f18099e83a1/konnectivity-agent/0.log"
Sep 09 22:18:09.093961 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:09.093882    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-221.ec2.internal_a1378741cd49bb53a4187c0b23b12755/haproxy/0.log"
Sep 09 22:18:13.924774 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:13.924750    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pnqnf_7c6d4ad7-7888-4c91-8dc2-ecff50cdc8b2/kuadrant-console-plugin/0.log"
Sep 09 22:18:14.139107 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:14.139064    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-sp7fz_7167994c-bafe-4e71-81df-570f550e40a4/manager/0.log"
Sep 09 22:18:15.471821 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.471794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c/alertmanager/0.log"
Sep 09 22:18:15.492473 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.492439    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c/config-reloader/0.log"
Sep 09 22:18:15.525553 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.525524    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c/kube-rbac-proxy-web/0.log"
Sep 09 22:18:15.542381 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.542310    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c/kube-rbac-proxy/0.log"
Sep 09 22:18:15.561782 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.561749    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c/kube-rbac-proxy-metric/0.log"
Sep 09 22:18:15.582089 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.582062    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c/prom-label-proxy/0.log"
Sep 09 22:18:15.608764 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.608738    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9aa8dee8-61a9-4e2a-a7d4-8fc41fbfa01c/init-config-reloader/0.log"
Sep 09 22:18:15.671573 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.671538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mh96n_31b6d756-4bd2-4f8b-8abc-8fbc000a5329/kube-state-metrics/0.log"
Sep 09 22:18:15.692097 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.692071    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mh96n_31b6d756-4bd2-4f8b-8abc-8fbc000a5329/kube-rbac-proxy-main/0.log"
Sep 09 22:18:15.719031 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.719004    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mh96n_31b6d756-4bd2-4f8b-8abc-8fbc000a5329/kube-rbac-proxy-self/0.log"
Sep 09 22:18:15.750423 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.750399    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bb65f78d5-2g9dl_d2b24c70-d39c-40c7-ad04-8e3d4de8d35b/metrics-server/0.log"
Sep 09 22:18:15.876351 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.876330    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnj2k_c136aec1-2cab-45a6-aacc-141254796986/node-exporter/0.log"
Sep 09 22:18:15.894832 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.894812    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnj2k_c136aec1-2cab-45a6-aacc-141254796986/kube-rbac-proxy/0.log"
Sep 09 22:18:15.918732 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:15.918712    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnj2k_c136aec1-2cab-45a6-aacc-141254796986/init-textfile/0.log"
Sep 09 22:18:16.022787 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.022757    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fp486_4441a240-ee74-4b9f-906c-318581bb2864/kube-rbac-proxy-main/0.log"
Sep 09 22:18:16.046431 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.046409    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fp486_4441a240-ee74-4b9f-906c-318581bb2864/kube-rbac-proxy-self/0.log"
Sep 09 22:18:16.084003 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.083974    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fp486_4441a240-ee74-4b9f-906c-318581bb2864/openshift-state-metrics/0.log"
Sep 09 22:18:16.162978 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.162948    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d3969580-afdb-4791-9790-5cddd3e6d160/prometheus/0.log"
Sep 09 22:18:16.191134 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.191108    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d3969580-afdb-4791-9790-5cddd3e6d160/config-reloader/0.log"
Sep 09 22:18:16.215034 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.215012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d3969580-afdb-4791-9790-5cddd3e6d160/thanos-sidecar/0.log"
Sep 09 22:18:16.238423 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.238398    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d3969580-afdb-4791-9790-5cddd3e6d160/kube-rbac-proxy-web/0.log"
Sep 09 22:18:16.268250 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.268223    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d3969580-afdb-4791-9790-5cddd3e6d160/kube-rbac-proxy/0.log"
Sep 09 22:18:16.295203 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.295175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d3969580-afdb-4791-9790-5cddd3e6d160/kube-rbac-proxy-thanos/0.log"
Sep 09 22:18:16.321934 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:16.321905    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d3969580-afdb-4791-9790-5cddd3e6d160/init-config-reloader/0.log"
Sep 09 22:18:19.635169 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.635127    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"]
Sep 09 22:18:19.635579 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.635553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy"
Sep 09 22:18:19.635579 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.635565    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d29cb97-241e-492f-92fb-d7702847c2f6" containerName="istio-proxy"
Sep 09 22:18:19.638815 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.638789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.641096 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.641017    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjhm4\"/\"kube-root-ca.crt\""
Sep 09 22:18:19.641096 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.641017    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pjhm4\"/\"default-dockercfg-627h7\""
Sep 09 22:18:19.641888 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.641868    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjhm4\"/\"openshift-service-ca.crt\""
Sep 09 22:18:19.648455 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.648430    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"]
Sep 09 22:18:19.739529 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.739489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-proc\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.739529 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.739530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-sys\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.739806 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.739647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-lib-modules\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.739806 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.739673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-podres\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.739806 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.739739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbqfm\" (UniqueName: \"kubernetes.io/projected/949e79af-21e3-4a10-9223-6fda3c01ebf8-kube-api-access-rbqfm\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840369 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-proc\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840369 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-sys\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840628 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-lib-modules\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840628 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-podres\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840628 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-proc\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840628 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqfm\" (UniqueName: \"kubernetes.io/projected/949e79af-21e3-4a10-9223-6fda3c01ebf8-kube-api-access-rbqfm\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840628 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-sys\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840628 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-podres\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.840858 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.840630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/949e79af-21e3-4a10-9223-6fda3c01ebf8-lib-modules\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.848482 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.848456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbqfm\" (UniqueName: \"kubernetes.io/projected/949e79af-21e3-4a10-9223-6fda3c01ebf8-kube-api-access-rbqfm\") pod \"perf-node-gather-daemonset-clczd\" (UID: \"949e79af-21e3-4a10-9223-6fda3c01ebf8\") " pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:19.949674 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:19.949564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:20.225823 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:20.223140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"]
Sep 09 22:18:20.227823 ip-10-0-139-221 kubenswrapper[2579]: W0909 22:18:20.227787    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod949e79af_21e3_4a10_9223_6fda3c01ebf8.slice/crio-30ad9ab46e9328f4ae32fd2e013d3603380336c66159c1be89a3d0d8aa51e3c9 WatchSource:0}: Error finding container 30ad9ab46e9328f4ae32fd2e013d3603380336c66159c1be89a3d0d8aa51e3c9: Status 404 returned error can't find the container with id 30ad9ab46e9328f4ae32fd2e013d3603380336c66159c1be89a3d0d8aa51e3c9
Sep 09 22:18:20.745161 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:20.745119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd" event={"ID":"949e79af-21e3-4a10-9223-6fda3c01ebf8","Type":"ContainerStarted","Data":"a78edbd5b85e9da647fa4522be177b0690fa93e73ae930ba45ad378a3d1ec28e"}
Sep 09 22:18:20.745619 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:20.745190    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:20.745619 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:20.745216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd" event={"ID":"949e79af-21e3-4a10-9223-6fda3c01ebf8","Type":"ContainerStarted","Data":"30ad9ab46e9328f4ae32fd2e013d3603380336c66159c1be89a3d0d8aa51e3c9"}
Sep 09 22:18:20.762023 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:20.761973    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd" podStartSLOduration=1.761956047 podStartE2EDuration="1.761956047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:18:20.760066284 +0000 UTC m=+2375.182428983" watchObservedRunningTime="2026-09-09 22:18:20.761956047 +0000 UTC m=+2375.184318747"
Sep 09 22:18:21.137987 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:21.137954    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v99wb_d834a708-778c-44a5-ad5a-ecdcb56aa53f/dns/0.log"
Sep 09 22:18:21.157579 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:21.157551    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v99wb_d834a708-778c-44a5-ad5a-ecdcb56aa53f/kube-rbac-proxy/0.log"
Sep 09 22:18:21.283020 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:21.282980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vg5zh_f86d9c73-bc49-4713-96dd-4b96e1cf9156/dns-node-resolver/0.log"
Sep 09 22:18:21.815478 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:21.815446    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d8dsb_9bbe1e6f-07bc-4d52-b391-79178b7b6be9/node-ca/0.log"
Sep 09 22:18:22.737575 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:22.737535    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9_bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4/istio-proxy/0.log"
Sep 09 22:18:23.040051 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:23.039953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7vd78_df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc/istio-proxy/0.log"
Sep 09 22:18:23.048829 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:23.048800    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7vd78_df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc/istio-proxy/1.log"
Sep 09 22:18:23.079537 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:23.079495    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6jbpm_6e710c3b-e23c-4789-ba15-0088d6d84359/payload-pre-processing/0.log"
Sep 09 22:18:23.149946 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:23.149917    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c55bbf95b-wdt7z_5613aec3-899d-4cbf-a406-7e1d4b0f5bd7/router/0.log"
Sep 09 22:18:24.019478 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:24.019416    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mfrlp_667ff566-22cb-4cdc-beb3-6bab86102ec5/serve-healthcheck-canary/0.log"
Sep 09 22:18:24.505310 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:24.505260    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-6z657_c0aa425c-e7ed-4feb-9610-55b9cf1ee45e/insights-operator/0.log"
Sep 09 22:18:24.506010 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:24.505982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-6z657_c0aa425c-e7ed-4feb-9610-55b9cf1ee45e/insights-operator/1.log"
Sep 09 22:18:24.658656 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:24.658618    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfcbw_92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3/kube-rbac-proxy/0.log"
Sep 09 22:18:24.677990 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:24.677966    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfcbw_92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3/exporter/0.log"
Sep 09 22:18:24.699616 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:24.699564    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfcbw_92e50cc2-321e-4ddc-bf0a-c0cbcb7df1e3/extractor/0.log"
Sep 09 22:18:26.760844 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:26.760818    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pjhm4/perf-node-gather-daemonset-clczd"
Sep 09 22:18:26.813849 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:26.813819    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ndh64_bb9f6ae7-1d80-4c29-899f-0f935069a4ab/manager/0.log"
Sep 09 22:18:27.012749 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:27.012627    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jmpp8_478bfb26-f142-4867-a37e-089756d3809b/manager/0.log"
Sep 09 22:18:27.336678 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:27.336578    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56bc947947-gwfqd_81f4008d-743b-4c71-a8b6-3c24545ab2a8/manager/0.log"
Sep 09 22:18:27.354620 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:27.354574    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56bc947947-gwfqd_81f4008d-743b-4c71-a8b6-3c24545ab2a8/copy-manifests/0.log"
Sep 09 22:18:28.630224 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:28.630175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-798b8d4d4-cswmr_a3b1a73d-66d5-4ada-8228-c4e682064f5f/manager/0.log"
Sep 09 22:18:29.554137 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:29.554097    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9_bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4/istio-proxy/0.log"
Sep 09 22:18:29.821134 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:29.821102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7vd78_df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc/istio-proxy/0.log"
Sep 09 22:18:29.829196 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:29.829172    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7vd78_df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc/istio-proxy/1.log"
Sep 09 22:18:29.853687 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:29.853661    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6jbpm_6e710c3b-e23c-4789-ba15-0088d6d84359/payload-pre-processing/0.log"
Sep 09 22:18:29.905047 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:29.905020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c55bbf95b-wdt7z_5613aec3-899d-4cbf-a406-7e1d4b0f5bd7/router/0.log"
Sep 09 22:18:31.482946 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:31.482913    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7m2p9_bf48a25a-3a8a-4177-944f-e1ee5d9ef8c4/istio-proxy/0.log"
Sep 09 22:18:31.758900 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:31.758813    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7vd78_df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc/istio-proxy/0.log"
Sep 09 22:18:31.767529 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:31.767506    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7vd78_df87cceb-f7da-4e26-bfcc-5c3fd4fcbfbc/istio-proxy/1.log"
Sep 09 22:18:31.792257 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:31.792230    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6jbpm_6e710c3b-e23c-4789-ba15-0088d6d84359/payload-pre-processing/0.log"
Sep 09 22:18:31.851428 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:31.851403    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c55bbf95b-wdt7z_5613aec3-899d-4cbf-a406-7e1d4b0f5bd7/router/0.log"
Sep 09 22:18:33.980535 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:33.980504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-cl7b6_9141d428-585e-4ca7-8aa9-7ccc8250b425/migrator/0.log"
Sep 09 22:18:34.003432 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:34.003399    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-cl7b6_9141d428-585e-4ca7-8aa9-7ccc8250b425/graceful-termination/0.log"
Sep 09 22:18:34.355906 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:34.355870    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-25d5s_598d76a6-65f4-48d0-8deb-96c1db8c084a/kube-storage-version-migrator-operator/1.log"
Sep 09 22:18:34.356732 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:34.356707    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-25d5s_598d76a6-65f4-48d0-8deb-96c1db8c084a/kube-storage-version-migrator-operator/0.log"
Sep 09 22:18:35.543239 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.543208    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7k8zp_8db25aa0-8917-4c9c-8d86-855d010e4c13/kube-multus-additional-cni-plugins/0.log"
Sep 09 22:18:35.561206 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.561176    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7k8zp_8db25aa0-8917-4c9c-8d86-855d010e4c13/egress-router-binary-copy/0.log"
Sep 09 22:18:35.579213 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.579186    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7k8zp_8db25aa0-8917-4c9c-8d86-855d010e4c13/cni-plugins/0.log"
Sep 09 22:18:35.597428 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.597400    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7k8zp_8db25aa0-8917-4c9c-8d86-855d010e4c13/bond-cni-plugin/0.log"
Sep 09 22:18:35.618022 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.617995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7k8zp_8db25aa0-8917-4c9c-8d86-855d010e4c13/routeoverride-cni/0.log"
Sep 09 22:18:35.636412 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.636390    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7k8zp_8db25aa0-8917-4c9c-8d86-855d010e4c13/whereabouts-cni-bincopy/0.log"
Sep 09 22:18:35.655094 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.655075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7k8zp_8db25aa0-8917-4c9c-8d86-855d010e4c13/whereabouts-cni/0.log"
Sep 09 22:18:35.901455 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.901421    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lqrz5_d166fa40-a07a-4ca2-9909-37ca14b614b3/kube-multus/0.log"
Sep 09 22:18:35.966682 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.966653    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wvv59_048fbca3-d90e-4740-a1d0-1b7f77453e04/network-metrics-daemon/0.log"
Sep 09 22:18:35.981083 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:35.981052    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wvv59_048fbca3-d90e-4740-a1d0-1b7f77453e04/kube-rbac-proxy/0.log"
Sep 09 22:18:37.050544 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:37.050511    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjtjx_75434aec-7a98-4360-b03d-867428a3e8ae/ovn-controller/0.log"
Sep 09 22:18:37.067469 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:37.067443    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjtjx_75434aec-7a98-4360-b03d-867428a3e8ae/ovn-acl-logging/0.log"
Sep 09 22:18:37.093455 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:37.093426    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjtjx_75434aec-7a98-4360-b03d-867428a3e8ae/kube-rbac-proxy-node/0.log"
Sep 09 22:18:37.112707 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:37.112680    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjtjx_75434aec-7a98-4360-b03d-867428a3e8ae/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 22:18:37.128420 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:37.128394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjtjx_75434aec-7a98-4360-b03d-867428a3e8ae/northd/0.log"
Sep 09 22:18:37.148568 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:37.148538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjtjx_75434aec-7a98-4360-b03d-867428a3e8ae/nbdb/0.log"
Sep 09 22:18:37.168454 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:37.168426    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjtjx_75434aec-7a98-4360-b03d-867428a3e8ae/sbdb/0.log"
Sep 09 22:18:37.230732 ip-10-0-139-221 kubenswrapper[2579]: I0909 22:18:37.230701    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjtjx_75434aec-7a98-4360-b03d-867428a3e8ae/ovnkube-controller/0.log"