Jul 30 03:32:47.320497 ip-10-0-137-125 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 30 03:32:47.320513 ip-10-0-137-125 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 30 03:32:47.320522 ip-10-0-137-125 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 30 03:32:47.320843 ip-10-0-137-125 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 30 03:32:57.465446 ip-10-0-137-125 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 30 03:32:57.465465 ip-10-0-137-125 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c8579c2fd89c450db2390468a210271f --
Jul 30 03:34:51.922673 ip-10-0-137-125 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 03:34:52.305651 ip-10-0-137-125 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 03:34:52.305651 ip-10-0-137-125 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 03:34:52.305651 ip-10-0-137-125 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 03:34:52.305651 ip-10-0-137-125 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 03:34:52.305651 ip-10-0-137-125 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 03:34:52.307157 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.307068    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 03:34:52.311733 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311717    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:34:52.311733 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311734    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311739    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311742    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311745    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311749    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311752    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311754    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311758    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311761    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311764    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311768    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311771    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311775    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311779    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311783    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311785    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311788    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311791    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311793    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:34:52.311796 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311796    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311799    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311802    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311805    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311808    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311810    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311813    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311816    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311818    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311821    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311823    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311826    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311828    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311831    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311833    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311836    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311838    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311841    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311843    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311846    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:34:52.312288 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311848    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311851    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311861    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311864    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311866    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311868    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311871    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311873    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311876    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311884    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311886    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311889    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311891    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311893    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311897    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311900    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311902    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311905    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311907    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311910    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:34:52.312784 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311912    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311915    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311918    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311920    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311922    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311925    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311927    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311930    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311934    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311937    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311940    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311943    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311946    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311949    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311951    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311959    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311962    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311965    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311968    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311970    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:34:52.313281 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311972    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311976    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311978    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311981    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311983    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311988    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311991    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311994    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.311997    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312000    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312003    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312006    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312009    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312011    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312014    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312016    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312019    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312021    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312024    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:34:52.313791 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312788    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312796    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312799    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312802    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312804    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312807    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312810    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312812    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312815    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312821    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312824    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312827    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312829    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312832    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312835    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312837    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312840    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312842    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312845    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312849    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:34:52.314255 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312852    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312855    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312858    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312863    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312866    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312870    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312873    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312876    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312879    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312882    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312885    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312888    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312890    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312893    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312896    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312899    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312901    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312904    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312906    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:34:52.314751 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312909    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312911    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312914    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312917    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312919    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312922    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312925    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312927    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312930    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312932    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312935    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312937    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312941    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312944    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312948    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312951    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312954    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312957    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312960    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:34:52.315224 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312963    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312965    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312968    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312971    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312973    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312977    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312979    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312982    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312984    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312987    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312989    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312992    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312995    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.312997    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313000    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313002    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313005    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313008    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313010    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313013    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:34:52.315694 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313016    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313018    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313020    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313023    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313026    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313029    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313031    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313034    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313036    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313039    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313042    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313044    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313047    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313050    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313053    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313056    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313059    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313061    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313063    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313066    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:34:52.316207 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313068    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313876    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313883    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313886    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313889    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313892    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313895    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313898    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313901    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313905    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313908    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313911    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313914    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313917    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313921    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313925    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313928    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313931    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313934    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313937    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:34:52.316713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313940    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313943    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313945    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313948    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313951    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313953    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313956    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313958    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313961    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313964    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313966    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313969    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313971    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313974    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313977    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313979    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313982    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313984    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313989    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313992    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:34:52.317198 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313994    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.313997    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314000    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314002    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314005    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314007    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314010    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314012    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314015    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314018    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314021    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314025    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314028    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314032    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314034    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314037    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314039    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314042    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314044    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:34:52.317749 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314047    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314050    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314052    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314055    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314057    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314060    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314062    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314065    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314067    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314070    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314072    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314076    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314078    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314081    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314083    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314086    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314089    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314092    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314096    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:34:52.318218 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314100    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314103    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314106    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314111    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314114    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314116    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314119    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314121    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314124    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314126    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314129    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314131    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314134    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314136    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314139    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314142    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314144    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314147    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314150    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:34:52.318720 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314152    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314155    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314158    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314227    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314235    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314245    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314250    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314254    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314258    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314263    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314268    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314272    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314275    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314279    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314282    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314285    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314289    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314292    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314295    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314298    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314301    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314305    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314308    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 03:34:52.319185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314311    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314314    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314318    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314322    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314325    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314328    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314331    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314334    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314337    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314340    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314344    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314346    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314351    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314354    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314357    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314360    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314363    2560 flags.go:64] FLAG: --enable-server="true"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314366    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314370    2560 flags.go:64] FLAG: --event-burst="100"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314374    2560 flags.go:64] FLAG: --event-qps="50"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314376    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314385    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314388    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314392    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 03:34:52.319746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314395    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314398    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314401    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314404    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314407    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314410    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314413    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314416    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314418    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314421    2560 flags.go:64] FLAG: --feature-gates=""
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314425    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314428    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314431    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314434    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314437    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314439    2560 flags.go:64] FLAG: --help="false"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314442    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-137-125.ec2.internal"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314445    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314448    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314451    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314454    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314457    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314460    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 03:34:52.320324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314464    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314467    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314469    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314472    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314475    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314478    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314481    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314491    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314495    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314497    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314500    2560 flags.go:64] FLAG: --lock-file=""
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314503    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314506    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314508    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314518    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314534    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314537    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314539    2560 flags.go:64] FLAG: --logging-format="text"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314542    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314546    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314548    2560 flags.go:64] FLAG: --manifest-url=""
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314551    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314556    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314559    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314562    2560 flags.go:64] FLAG: --max-pods="110"
Jul 30 03:34:52.321249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314565    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314568    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314572    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314575    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314578    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314580    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314584    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314591    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314594    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314597    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314599    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314602    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314607    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314610    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314613    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314622    2560 flags.go:64] FLAG: --port="10250"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314625    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314628    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ab685f2adbed80bc"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314631    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314634    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314636    2560 flags.go:64] FLAG: --register-node="true"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314639    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314643    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314646    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 30 03:34:52.322352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314648    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314651    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314655    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314658    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314661    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314663    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314666    2560 flags.go:64] FLAG: --runonce="false"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314669    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314672    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314675    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314677    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314680    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314683    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314686    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314689    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314692    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314696    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314699    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314702    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314705    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314708    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314710    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314716    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314719    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314727    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 03:34:52.323394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314731    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314734    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314737    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314740    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314742    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314745    2560 flags.go:64] FLAG: --v="2"
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314749    2560 flags.go:64] FLAG: --version="false"
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314753    2560 flags.go:64] FLAG: --vmodule=""
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314757    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.314760    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314895    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314899    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314902    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314905    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314908    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314910    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314913    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314915    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314918    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314921    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314923    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314926    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:34:52.324480 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314930    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314934    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314937    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314940    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314942    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314945    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314947    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314950    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314952    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314955    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314958    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314961    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314963    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314966    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314968    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314971    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314973    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314976    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314978    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314981    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:34:52.325264 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314983    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314986    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314988    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314991    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314994    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314997    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.314999    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315001    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315004    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315007    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315009    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315012    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315016    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315020    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315022    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315025    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315027    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315030    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315033    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315035    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:34:52.326027 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315038    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315041    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315044    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315047    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315049    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315052    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315054    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315057    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315060    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315062    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315066    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315070    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315073    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315076    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315079    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315082    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315085    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315088    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315092    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:34:52.326584 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315095    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315098    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315100    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315103    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315105    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315110    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315114    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315116    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315119    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315122    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315124    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315127    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315129    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315133    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315137    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315140    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315143    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315152    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315155    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:34:52.327133 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315158    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315161    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315164    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315167    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315170    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315173    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315175    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315178    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.315181    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.315745    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.322849    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.322869    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.322888    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.322901    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.322986    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:34:52.327738 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.322993    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.322998    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323002    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323007    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323011    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323016    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323020    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323025    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323029    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323034    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323038    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323042    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323047    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323053    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323058    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323064    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323069    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323074    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323078    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:34:52.328249 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323082    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323086    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323091    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323095    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323099    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323104    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323109    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323113    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323117    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323123    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323127    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323132    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323136    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323140    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323145    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323149    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323153    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323157    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323161    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323165    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:34:52.328776 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323169    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323173    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323177    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323181    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323185    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323190    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323194    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323198    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323202    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323207    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323211    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323215    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323219    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323226    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323231    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323238    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323243    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323248    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323253    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:34:52.329334 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323257    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323263    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323268    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323273    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323278    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323282    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323286    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323290    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323295    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323299    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323303    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323307    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323311    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323316    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323320    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323325    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323329    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323333    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323337    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323341    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:34:52.329994 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323346    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323350    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323354    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323359    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323364    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323368    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323372    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323376    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323380    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323384    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323389    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323392    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323396    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323400    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323405    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323409    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323412    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323416    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323421    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:34:52.330497 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323425    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.323433    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323631    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323641    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323646    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323650    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323656    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323660    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323664    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323669    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323675    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323683    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323689    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323694    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:34:52.331013 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323699    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323704    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323708    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323714    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323719    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323725    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323729    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323734    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323738    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323742    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323747    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323751    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323755    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323760    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323764    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323768    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323773    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323777    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323781    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323785    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:34:52.331500 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323790    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323794    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323798    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323802    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323806    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323811    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323815    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323819    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323823    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323827    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323831    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323835    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323839    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323844    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323848    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323853    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323858    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323863    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323867    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323871    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:34:52.332059 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323876    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323880    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323884    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323888    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323893    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323897    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323902    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323906    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323910    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323916    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323922    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323926    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323930    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323935    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323939    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323943    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323947    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323951    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323955    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:34:52.332718 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323960    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323964    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323968    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323972    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323976    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323980    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323984    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323988    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323992    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.323996    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324001    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324005    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324009    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324013    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324017    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324022    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324025    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324030    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324034    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:34:52.333270 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324038    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324042    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324046    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324050    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324054    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324058    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324062    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324067    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:52.324071    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.324079    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.324242    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.327924    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.329107    2560 server.go:1019] "Starting client certificate rotation"
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.329218    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 03:34:52.333901 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.329250    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 03:34:52.334295 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.334248    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 03:34:52.335569 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.335551    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 03:34:52.345931 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.345910    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 30 03:34:52.347937 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.347923    2560 log.go:25] "Validated CRI v1 image API"
Jul 30 03:34:52.348665 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.348652    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 03:34:52.353337 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.353314    2560 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b287f122-5d13-4b36-9299-1df8d9b5b893:/dev/nvme0n1p3 c17274dd-1a2d-42e6-8c1b-ce91499228fa:/dev/nvme0n1p4]
Jul 30 03:34:52.353424 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.353335    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 03:34:52.359271 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.359150    2560 manager.go:216] Machine: {Timestamp:2026-07-30 03:34:52.357951626 +0000 UTC m=+0.328297137 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119600 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea0b1f511ec8e98f8ffb06ef821c9 SystemUUID:ec2ea0b1-f511-ec8e-98f8-ffb06ef821c9 BootID:c8579c2f-d89c-450d-b239-0468a210271f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:a5:a9:4a:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:a5:a9:4a:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:99:7c:e5:d4:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 03:34:52.359271 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.359260    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 03:34:52.359442 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.359373    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 03:34:52.359671 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.359656    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 03:34:52.362059 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.362030    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 03:34:52.362199 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.362062    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 03:34:52.362647 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.362637    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 03:34:52.362686 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.362649    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 03:34:52.362686 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.362657    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 03:34:52.363163 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.363153    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 03:34:52.363868 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.363857    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 03:34:52.365645 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.365634    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 03:34:52.365768 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.365759    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 03:34:52.365811 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.365788    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 03:34:52.365811 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.365798    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 03:34:52.365895 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.365817    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 03:34:52.365895 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.365825    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 30 03:34:52.365895 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.365848    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 03:34:52.366303 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.366288    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 03:34:52.366364 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.366322    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 03:34:52.368553 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.368538    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 03:34:52.371761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.371745    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 03:34:52.371853 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.371776    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 03:34:52.373297 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373282    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373300    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373306    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373312    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373320    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373326    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373334    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373339    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373346    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373351    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 03:34:52.373359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373360    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 03:34:52.373659 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.373366    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 03:34:52.374017 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.374006    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 03:34:52.374052 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.374018    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 03:34:52.374620 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.374604    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vr5n6"
Jul 30 03:34:52.375032 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:52.375006    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 30 03:34:52.375081 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:52.375006    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 30 03:34:52.376604 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.376590    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 30 03:34:52.377245 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.377234    2560 server.go:1268] "Started kubelet"
Jul 30 03:34:52.377384 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.377354    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 03:34:52.377487 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.377378    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 03:34:52.377487 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.377452    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 03:34:52.378217 ip-10-0-137-125 systemd[1]: Started Kubernetes Kubelet.
Jul 30 03:34:52.378968 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.378951    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 03:34:52.379061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.379025    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 03:34:52.381860 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.381838    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vr5n6"
Jul 30 03:34:52.382452 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:52.382429    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 03:34:52.386871 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.386856    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 03:34:52.386871 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.386864    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 03:34:52.387037 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:52.386994    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:52.387536 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.387497    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 03:34:52.388152 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388135    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 03:34:52.388221 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388155    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 03:34:52.388284 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388266    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 03:34:52.388331 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388285    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 03:34:52.388642 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388625    2560 factory.go:55] Registering systemd factory
Jul 30 03:34:52.388642 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388644    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 30 03:34:52.388869 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388850    2560 factory.go:153] Registering CRI-O factory
Jul 30 03:34:52.388940 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388873    2560 factory.go:223] Registration of the crio container factory successfully
Jul 30 03:34:52.388996 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388938    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 03:34:52.388996 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388969    2560 factory.go:103] Registering Raw factory
Jul 30 03:34:52.388996 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.388987    2560 manager.go:1196] Started watching for new ooms in manager
Jul 30 03:34:52.389341 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.389319    2560 manager.go:349] Starting recovery of all containers
Jul 30 03:34:52.391265 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.391229    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:34:52.395069 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:52.394904    2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-125.ec2.internal\" not found" node="ip-10-0-137-125.ec2.internal"
Jul 30 03:34:52.399322 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.399300    2560 manager.go:354] Recovery completed
Jul 30 03:34:52.404902 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.404888    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:34:52.406998 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.406978    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:34:52.407096 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.407017    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:34:52.407096 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.407029    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:34:52.407580 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.407556    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 03:34:52.407580 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.407581    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 03:34:52.407702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.407557    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:34:52.407702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.407601    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 03:34:52.408390 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.408375    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:34:52.408434 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.408404    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:34:52.408434 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.408414    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:34:52.409145 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.409132    2560 policy_none.go:49] "None policy: Start"
Jul 30 03:34:52.409192 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.409167    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 03:34:52.409192 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.409178    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 03:34:52.410130 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:52.410119    2560 policy_none.go:47] "Start"
Jul 30 03:34:53.465982 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.465955    2560 manager.go:337] "Starting Device Plugin manager"
Jul 30 03:34:53.465982 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.465987    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 03:34:53.466163 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.465998    2560 server.go:87] "Starting device plugin registration server"
Jul 30 03:34:53.466266 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.466254    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 03:34:53.466313 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.466270    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 03:34:53.466371 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.466354    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 03:34:53.466455 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.466443    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 03:34:53.466507 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.466456    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 03:34:53.467025 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.467007    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 03:34:53.467106 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.467064    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:53.566725 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.566690    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:34:53.568890 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.567914    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:34:53.568890 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.567949    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:34:53.568890 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.567959    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:34:53.568890 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.567987    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.570506 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.570482    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 03:34:53.571733 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.571691    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 03:34:53.571733 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.571727    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 03:34:53.571854 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.571748    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 03:34:53.571854 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.571783    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 03:34:53.574070 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.574045    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:34:53.577731 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.577717    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.577805 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.577737    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-125.ec2.internal\": node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:53.619282 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.619256    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:53.672367 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.672324    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal"]
Jul 30 03:34:53.672503 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.672427    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:34:53.673463 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.673447    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:34:53.673531 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.673477    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:34:53.673531 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.673488    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:34:53.675838 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.675825    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:34:53.675956 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.675929    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.675992 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.675972    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:34:53.676581 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.676566    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:34:53.676663 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.676593    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:34:53.676663 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.676602    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:34:53.676663 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.676569    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:34:53.676663 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.676664    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:34:53.676801 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.676676    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:34:53.678817 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.678804    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.678872 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.678829    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:34:53.681035 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.681017    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:34:53.681124 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.681049    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:34:53.681124 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.681064    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:34:53.705599 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.705574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f061476c9cab299179308efd01fc989-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal\" (UID: \"4f061476c9cab299179308efd01fc989\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.705599 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.705603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f061476c9cab299179308efd01fc989-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal\" (UID: \"4f061476c9cab299179308efd01fc989\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.705782 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.705620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8d00ccf9cbba644f9ba54a54e77d171e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-125.ec2.internal\" (UID: \"8d00ccf9cbba644f9ba54a54e77d171e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.713094 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.713070    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-125.ec2.internal\" not found" node="ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.717605 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.717588    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-125.ec2.internal\" not found" node="ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.719625 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.719610    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:53.806676 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.806646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8d00ccf9cbba644f9ba54a54e77d171e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-125.ec2.internal\" (UID: \"8d00ccf9cbba644f9ba54a54e77d171e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.806676 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.806681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f061476c9cab299179308efd01fc989-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal\" (UID: \"4f061476c9cab299179308efd01fc989\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.806878 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.806705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f061476c9cab299179308efd01fc989-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal\" (UID: \"4f061476c9cab299179308efd01fc989\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.806878 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.806737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8d00ccf9cbba644f9ba54a54e77d171e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-125.ec2.internal\" (UID: \"8d00ccf9cbba644f9ba54a54e77d171e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.806878 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.806761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f061476c9cab299179308efd01fc989-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal\" (UID: \"4f061476c9cab299179308efd01fc989\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.806878 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:53.806782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f061476c9cab299179308efd01fc989-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal\" (UID: \"4f061476c9cab299179308efd01fc989\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:53.819744 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.819714    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:53.920598 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:53.920517    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:54.017039 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.016997    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:54.020421 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.020403    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:54.020660 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.020639    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:54.121724 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.121686    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:54.222229 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.222160    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-125.ec2.internal\" not found"
Jul 30 03:34:54.238852 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.238828    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:34:54.302814 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.302777    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:54.313449 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.313427    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 03:34:54.314998 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.314983    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal"
Jul 30 03:34:54.323322 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.323297    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 03:34:54.344654 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.344630    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 03:34:54.344772 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.344754    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 03:34:54.344833 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.344758    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 03:34:54.344833 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.344759    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 03:34:54.369656 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.369632    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:34:54.381959 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.381940    2560 apiserver.go:52] "Watching apiserver"
Jul 30 03:34:54.386263 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.386247    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 03:34:54.386575 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.386557    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-fltkz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal","openshift-multus/multus-22d4x","openshift-multus/multus-additional-cni-plugins-mlpxh","openshift-multus/network-metrics-daemon-pc47f","kube-system/konnectivity-agent-r5szt","openshift-cluster-node-tuning-operator/tuned-8m5mq","openshift-image-registry/node-ca-p2q9q","openshift-network-diagnostics/network-check-target-qxxxh","openshift-network-operator/iptables-alerter-n6q7v","openshift-ovn-kubernetes/ovnkube-node-nclz4","kube-system/global-pull-secret-syncer-xfcs5","kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"]
Jul 30 03:34:54.389321 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.389306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.392554 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.392529    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.393322 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.393298    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:34:54.393467 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.393358    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rkfsc\""
Jul 30 03:34:54.393562 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.393390    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 03:34:54.393726 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.393698    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.395094 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395071    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 03:34:54.395182 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395076    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 03:34:54.395182 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395101    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 03:34:54.395442 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395424    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 03:34:54.395442 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395426    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-spxkq\""
Jul 30 03:34:54.395589 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395502    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 03:34:54.395589 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395516    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 03:34:54.395779 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395764    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 03:34:54.395847 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395784    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 03:34:54.395847 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395792    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 03:34:54.395926 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395794    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-knrzr\""
Jul 30 03:34:54.396015 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.395998    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.396064 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.396030    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 03:34:54.398115 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.398096    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5pq5v\""
Jul 30 03:34:54.398308 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.398295    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:54.398395 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.398381    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 03:34:54.398431 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.398383    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:34:54.398431 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.398418    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 03:34:54.400636 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.400619    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:34:54.402142 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.402126    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 03:34:54.402142 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.402133    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vx6rm\""
Jul 30 03:34:54.402316 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.402150    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 03:34:54.402316 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.402174    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 03:29:53 +0000 UTC" deadline="2028-01-02 18:18:39.790223094 +0000 UTC"
Jul 30 03:34:54.402316 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.402215    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12518h43m45.388011379s"
Jul 30 03:34:54.402834 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.402821    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 03:34:54.402964 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.402948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:54.403045 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.403024    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:34:54.405453 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.405285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.406982 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.406957    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 03:34:54.406982 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.406961    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 03:34:54.407116 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.407001    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zllnd\""
Jul 30 03:34:54.407116 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.407058    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 03:34:54.407531 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.407519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.409336 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409321    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 03:34:54.409481 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409465    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8dk44\""
Jul 30 03:34:54.409541 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409527    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 03:34:54.409581 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409475    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 03:34:54.409711 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-hostroot\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.409776 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.409776 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovn-node-metrics-cert\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.409776 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:54.409919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-cni-bin\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.409919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409802    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.409919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-tuned\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.409919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-log-socket\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.409919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8xj\" (UniqueName: \"kubernetes.io/projected/ebf7b93f-53c0-4c89-aa84-d9188b09e918-kube-api-access-zk8xj\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.409919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-cnibin\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.409919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-system-cni-dir\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-slash\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-device-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.409988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-run\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410034    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-systemd\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1236a4cf-a822-4222-ab7b-ecec2387f240-konnectivity-ca\") pod \"konnectivity-agent-r5szt\" (UID: \"1236a4cf-a822-4222-ab7b-ecec2387f240\") " pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-sys\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p26mf\" (UniqueName: \"kubernetes.io/projected/da4390c0-f2af-453b-8000-3ddfc8ace76e-kube-api-access-p26mf\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-run-ovn-kubernetes\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovnkube-config\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410183    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-etc-kubernetes\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-os-release\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.410226 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da4390c0-f2af-453b-8000-3ddfc8ace76e-tmp\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-env-overrides\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-multus-certs\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5zp\" (UniqueName: \"kubernetes.io/projected/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-kube-api-access-5p5zp\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-var-lib-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8708fd8-0e95-41fb-914c-54b8f9828bd4-cni-binary-copy\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-conf-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cnibin\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-run-netns\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-etc-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-cni-netd\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysconfig\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-kubelet\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-ovn\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.410637 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-node-log\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410651    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-cni-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-os-release\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-netns\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-daemon-config\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs8w\" (UniqueName: \"kubernetes.io/projected/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-kube-api-access-nqs8w\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-modprobe-d\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovnkube-script-lib\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-socket-dir-parent\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-kubelet\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hz7\" (UniqueName: \"kubernetes.io/projected/f8708fd8-0e95-41fb-914c-54b8f9828bd4-kube-api-access-b5hz7\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-sys-fs\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410957    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysctl-conf\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.410981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-cni-multus\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-system-cni-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411227 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-registration-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411068    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysctl-d\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-host\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1236a4cf-a822-4222-ab7b-ecec2387f240-agent-certs\") pod \"konnectivity-agent-r5szt\" (UID: \"1236a4cf-a822-4222-ab7b-ecec2387f240\") " pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-systemd\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411201    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-lib-modules\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-var-lib-kubelet\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-systemd-units\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-kubernetes\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbdx\" (UniqueName: \"kubernetes.io/projected/704164d4-c34d-4ba7-8080-b59d40739cf9-kube-api-access-xzbdx\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-cni-bin\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-k8s-cni-cncf-io\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411404    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411432    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411436    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411410    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wt5kt\""
Jul 30 03:34:54.411790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.412388 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.411527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-socket-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.412388 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.412134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.412856 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.412841    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 03:34:54.413429 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.413416    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 03:34:54.413753 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.413737    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nl4nf\""
Jul 30 03:34:54.414010 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.413986    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 03:34:54.416336 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.416318    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.416411 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.416384    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:34:54.434281 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.434258    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7d5k6"
Jul 30 03:34:54.440740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.440720    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7d5k6"
Jul 30 03:34:54.503904 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.503887    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 03:34:54.512329 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.512444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-systemd\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.512444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-lib-modules\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.512444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-var-lib-kubelet\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.512444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-systemd-units\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.512444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/239554d7-aed4-4f52-8b1e-fdd8b3758c97-kubelet-config\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.512444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-systemd\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.512664 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512461    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-var-lib-kubelet\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.512664 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/239554d7-aed4-4f52-8b1e-fdd8b3758c97-dbus\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.512664 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512475    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-lib-modules\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.512664 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-systemd-units\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.512664 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-kubernetes\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzn27\" (UniqueName: \"kubernetes.io/projected/519e4a13-90e3-4d40-a097-cecb5171c54c-kube-api-access-mzn27\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512713    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbdx\" (UniqueName: \"kubernetes.io/projected/704164d4-c34d-4ba7-8080-b59d40739cf9-kube-api-access-xzbdx\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-kubernetes\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-cni-bin\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7d5\" (UniqueName: \"kubernetes.io/projected/48c8e946-a078-4b39-912c-fde7009abb28-kube-api-access-wk7d5\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-k8s-cni-cncf-io\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.512864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-cni-bin\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512901    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-socket-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-k8s-cni-cncf-io\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512928    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-hostroot\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-hostroot\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512962    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-socket-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.512984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovn-node-metrics-cert\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-cni-bin\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-tuned\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-log-socket\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8xj\" (UniqueName: \"kubernetes.io/projected/ebf7b93f-53c0-4c89-aa84-d9188b09e918-kube-api-access-zk8xj\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-cni-bin\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.513264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513141    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48c8e946-a078-4b39-912c-fde7009abb28-hosts-file\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-log-socket\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-cnibin\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-system-cni-dir\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513229    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-cnibin\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-slash\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1403882-8905-4bed-b140-58eb42b80794-host-slash\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-device-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513283    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-system-cni-dir\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513321    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-slash\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-device-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513356    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-run\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-systemd\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-run\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1236a4cf-a822-4222-ab7b-ecec2387f240-konnectivity-ca\") pod \"konnectivity-agent-r5szt\" (UID: \"1236a4cf-a822-4222-ab7b-ecec2387f240\") " pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-sys\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513443    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-systemd\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.514080 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513446    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513351    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p26mf\" (UniqueName: \"kubernetes.io/projected/da4390c0-f2af-453b-8000-3ddfc8ace76e-kube-api-access-p26mf\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-run-ovn-kubernetes\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-sys\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovnkube-config\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-run-ovn-kubernetes\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-etc-kubernetes\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-os-release\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da4390c0-f2af-453b-8000-3ddfc8ace76e-tmp\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-env-overrides\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdsj\" (UniqueName: \"kubernetes.io/projected/b1403882-8905-4bed-b140-58eb42b80794-kube-api-access-vrdsj\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513818    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-os-release\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-multus-certs\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.515033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513857    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5zp\" (UniqueName: \"kubernetes.io/projected/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-kube-api-access-5p5zp\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-var-lib-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-etc-kubernetes\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8708fd8-0e95-41fb-914c-54b8f9828bd4-cni-binary-copy\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-conf-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cnibin\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.513991    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-run-netns\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-etc-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-var-lib-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514086    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-etc-openvswitch\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.514095    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:55.014033651 +0000 UTC m=+1.968668323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514097    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-cnibin\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.513989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1236a4cf-a822-4222-ab7b-ecec2387f240-konnectivity-ca\") pod \"konnectivity-agent-r5szt\" (UID: \"1236a4cf-a822-4222-ab7b-ecec2387f240\") " pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovnkube-config\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.515804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-conf-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-run-netns\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-multus-certs\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-cni-netd\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/519e4a13-90e3-4d40-a097-cecb5171c54c-host\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-cni-netd\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysconfig\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-kubelet\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-ovn\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-node-log\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1403882-8905-4bed-b140-58eb42b80794-iptables-alerter-script\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/519e4a13-90e3-4d40-a097-cecb5171c54c-serviceca\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8708fd8-0e95-41fb-914c-54b8f9828bd4-cni-binary-copy\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48c8e946-a078-4b39-912c-fde7009abb28-tmp-dir\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-run-ovn\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.516317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-cni-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-kubelet\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-os-release\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysconfig\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-node-log\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-netns\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf7b93f-53c0-4c89-aa84-d9188b09e918-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-os-release\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-daemon-config\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-run-netns\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs8w\" (UniqueName: \"kubernetes.io/projected/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-kube-api-access-nqs8w\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-cni-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-modprobe-d\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovnkube-script-lib\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-socket-dir-parent\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514934    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-kubelet\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514961    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hz7\" (UniqueName: \"kubernetes.io/projected/f8708fd8-0e95-41fb-914c-54b8f9828bd4-kube-api-access-b5hz7\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-env-overrides\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.517123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.514986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-sys-fs\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysctl-conf\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-cni-multus\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-system-cni-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-modprobe-d\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-sys-fs\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-registration-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysctl-d\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-host\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515222    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysctl-conf\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1236a4cf-a822-4222-ab7b-ecec2387f240-agent-certs\") pod \"konnectivity-agent-r5szt\" (UID: \"1236a4cf-a822-4222-ab7b-ecec2387f240\") " pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515265    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-daemon-config\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-cni-multus\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-system-cni-dir\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-multus-socket-dir-parent\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515394    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/704164d4-c34d-4ba7-8080-b59d40739cf9-registration-dir\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.518082 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8708fd8-0e95-41fb-914c-54b8f9828bd4-host-var-lib-kubelet\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.518933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515465    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-sysctl-d\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da4390c0-f2af-453b-8000-3ddfc8ace76e-host\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.515559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovnkube-script-lib\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.518933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.516989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da4390c0-f2af-453b-8000-3ddfc8ace76e-tmp\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.517054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da4390c0-f2af-453b-8000-3ddfc8ace76e-etc-tuned\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.518933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.517087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebf7b93f-53c0-4c89-aa84-d9188b09e918-ovn-node-metrics-cert\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.518933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.517523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1236a4cf-a822-4222-ab7b-ecec2387f240-agent-certs\") pod \"konnectivity-agent-r5szt\" (UID: \"1236a4cf-a822-4222-ab7b-ecec2387f240\") " pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:34:54.522015 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.521995    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:34:54.522015 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.522016    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:34:54.522200 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.522028    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jcm8b for pod openshift-network-diagnostics/network-check-target-qxxxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:34:54.522200 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.522083    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b podName:6e46faea-9a53-4a19-83cd-c1fbde38fae9 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:55.022070339 +0000 UTC m=+1.976704991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jcm8b" (UniqueName: "kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b") pod "network-check-target-qxxxh" (UID: "6e46faea-9a53-4a19-83cd-c1fbde38fae9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:34:54.523896 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.523877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p26mf\" (UniqueName: \"kubernetes.io/projected/da4390c0-f2af-453b-8000-3ddfc8ace76e-kube-api-access-p26mf\") pod \"tuned-8m5mq\" (UID: \"da4390c0-f2af-453b-8000-3ddfc8ace76e\") " pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.525631 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.525604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8xj\" (UniqueName: \"kubernetes.io/projected/ebf7b93f-53c0-4c89-aa84-d9188b09e918-kube-api-access-zk8xj\") pod \"ovnkube-node-nclz4\" (UID: \"ebf7b93f-53c0-4c89-aa84-d9188b09e918\") " pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.526163 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.526125    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs8w\" (UniqueName: \"kubernetes.io/projected/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-kube-api-access-nqs8w\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:54.532664 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.532643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbdx\" (UniqueName: \"kubernetes.io/projected/704164d4-c34d-4ba7-8080-b59d40739cf9-kube-api-access-xzbdx\") pod \"aws-ebs-csi-driver-node-n82pc\" (UID: \"704164d4-c34d-4ba7-8080-b59d40739cf9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.532797 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.532771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hz7\" (UniqueName: \"kubernetes.io/projected/f8708fd8-0e95-41fb-914c-54b8f9828bd4-kube-api-access-b5hz7\") pod \"multus-22d4x\" (UID: \"f8708fd8-0e95-41fb-914c-54b8f9828bd4\") " pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.537087 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.537058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5zp\" (UniqueName: \"kubernetes.io/projected/30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd-kube-api-access-5p5zp\") pod \"multus-additional-cni-plugins-mlpxh\" (UID: \"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd\") " pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.616197 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/519e4a13-90e3-4d40-a097-cecb5171c54c-host\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.616395 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1403882-8905-4bed-b140-58eb42b80794-iptables-alerter-script\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.616395 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/519e4a13-90e3-4d40-a097-cecb5171c54c-serviceca\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.616395 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48c8e946-a078-4b39-912c-fde7009abb28-tmp-dir\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.616395 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.616395 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/239554d7-aed4-4f52-8b1e-fdd8b3758c97-kubelet-config\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.616395 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/519e4a13-90e3-4d40-a097-cecb5171c54c-host\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.616395 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/239554d7-aed4-4f52-8b1e-fdd8b3758c97-kubelet-config\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/239554d7-aed4-4f52-8b1e-fdd8b3758c97-dbus\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.616430    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzn27\" (UniqueName: \"kubernetes.io/projected/519e4a13-90e3-4d40-a097-cecb5171c54c-kube-api-access-mzn27\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:54.616500    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret podName:239554d7-aed4-4f52-8b1e-fdd8b3758c97 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:55.116479057 +0000 UTC m=+2.071113720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret") pod "global-pull-secret-syncer-xfcs5" (UID: "239554d7-aed4-4f52-8b1e-fdd8b3758c97") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7d5\" (UniqueName: \"kubernetes.io/projected/48c8e946-a078-4b39-912c-fde7009abb28-kube-api-access-wk7d5\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48c8e946-a078-4b39-912c-fde7009abb28-hosts-file\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48c8e946-a078-4b39-912c-fde7009abb28-tmp-dir\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1403882-8905-4bed-b140-58eb42b80794-host-slash\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616595    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/239554d7-aed4-4f52-8b1e-fdd8b3758c97-dbus\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616652    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1403882-8905-4bed-b140-58eb42b80794-host-slash\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdsj\" (UniqueName: \"kubernetes.io/projected/b1403882-8905-4bed-b140-58eb42b80794-kube-api-access-vrdsj\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.616740 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48c8e946-a078-4b39-912c-fde7009abb28-hosts-file\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.617093 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.616753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1403882-8905-4bed-b140-58eb42b80794-iptables-alerter-script\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.617093 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.617005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/519e4a13-90e3-4d40-a097-cecb5171c54c-serviceca\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.624479 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.624453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdsj\" (UniqueName: \"kubernetes.io/projected/b1403882-8905-4bed-b140-58eb42b80794-kube-api-access-vrdsj\") pod \"iptables-alerter-n6q7v\" (UID: \"b1403882-8905-4bed-b140-58eb42b80794\") " pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.624632 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.624613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzn27\" (UniqueName: \"kubernetes.io/projected/519e4a13-90e3-4d40-a097-cecb5171c54c-kube-api-access-mzn27\") pod \"node-ca-p2q9q\" (UID: \"519e4a13-90e3-4d40-a097-cecb5171c54c\") " pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.624671 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.624614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7d5\" (UniqueName: \"kubernetes.io/projected/48c8e946-a078-4b39-912c-fde7009abb28-kube-api-access-wk7d5\") pod \"node-resolver-fltkz\" (UID: \"48c8e946-a078-4b39-912c-fde7009abb28\") " pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.651980 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.651954    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f061476c9cab299179308efd01fc989.slice/crio-33cf82f3dc4000c6d7cae8092ac0abf44a54ff2f16e8120a2900556aa0c401d6 WatchSource:0}: Error finding container 33cf82f3dc4000c6d7cae8092ac0abf44a54ff2f16e8120a2900556aa0c401d6: Status 404 returned error can't find the container with id 33cf82f3dc4000c6d7cae8092ac0abf44a54ff2f16e8120a2900556aa0c401d6
Jul 30 03:34:54.655816 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.655802    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:34:54.713767 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.713740    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d00ccf9cbba644f9ba54a54e77d171e.slice/crio-d9f2b71b37e6b0780d1f8bca280ad43f16e95923569798b1245700f073134ad0 WatchSource:0}: Error finding container d9f2b71b37e6b0780d1f8bca280ad43f16e95923569798b1245700f073134ad0: Status 404 returned error can't find the container with id d9f2b71b37e6b0780d1f8bca280ad43f16e95923569798b1245700f073134ad0
Jul 30 03:34:54.720353 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.720332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8m5mq"
Jul 30 03:34:54.728001 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.727980    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4390c0_f2af_453b_8000_3ddfc8ace76e.slice/crio-e0c980047619975c6e362297b7333eeb3f71112114f018c5ce047fb4c22ceffd WatchSource:0}: Error finding container e0c980047619975c6e362297b7333eeb3f71112114f018c5ce047fb4c22ceffd: Status 404 returned error can't find the container with id e0c980047619975c6e362297b7333eeb3f71112114f018c5ce047fb4c22ceffd
Jul 30 03:34:54.737028 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.736987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:34:54.744737 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.744713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf7b93f_53c0_4c89_aa84_d9188b09e918.slice/crio-3cbc59f29dfc546b5002667a4475ebd4f0a9cf2a9a861ed0b04dcdbb03eb8fab WatchSource:0}: Error finding container 3cbc59f29dfc546b5002667a4475ebd4f0a9cf2a9a861ed0b04dcdbb03eb8fab: Status 404 returned error can't find the container with id 3cbc59f29dfc546b5002667a4475ebd4f0a9cf2a9a861ed0b04dcdbb03eb8fab
Jul 30 03:34:54.750077 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.750061    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22d4x"
Jul 30 03:34:54.758138 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.758103    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8708fd8_0e95_41fb_914c_54b8f9828bd4.slice/crio-40f91fed53de04e9d947ce7b3a0098af56c38e09ed4abe1926f0ae2a3409b189 WatchSource:0}: Error finding container 40f91fed53de04e9d947ce7b3a0098af56c38e09ed4abe1926f0ae2a3409b189: Status 404 returned error can't find the container with id 40f91fed53de04e9d947ce7b3a0098af56c38e09ed4abe1926f0ae2a3409b189
Jul 30 03:34:54.764678 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.764656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mlpxh"
Jul 30 03:34:54.772599 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.772571    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d5a13d_4f2c_4f63_8a1f_b85b3b7c0ccd.slice/crio-505cddf693632430775ac337e85121ba7c815a9938fb61a4aafc25cfedc4eab9 WatchSource:0}: Error finding container 505cddf693632430775ac337e85121ba7c815a9938fb61a4aafc25cfedc4eab9: Status 404 returned error can't find the container with id 505cddf693632430775ac337e85121ba7c815a9938fb61a4aafc25cfedc4eab9
Jul 30 03:34:54.784645 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.784628    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:34:54.792399 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.792373    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1236a4cf_a822_4222_ab7b_ecec2387f240.slice/crio-165d96b6f4ef3ce33ccf43d304647bd2b2738f0b54831d5b2fba4ba620083f78 WatchSource:0}: Error finding container 165d96b6f4ef3ce33ccf43d304647bd2b2738f0b54831d5b2fba4ba620083f78: Status 404 returned error can't find the container with id 165d96b6f4ef3ce33ccf43d304647bd2b2738f0b54831d5b2fba4ba620083f78
Jul 30 03:34:54.797347 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.797328    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc"
Jul 30 03:34:54.805115 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.805093    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704164d4_c34d_4ba7_8080_b59d40739cf9.slice/crio-88d245fb1280c2a414adafcfb28137e3d27dca9b83a1f1b7865ca4d7d00dd4cd WatchSource:0}: Error finding container 88d245fb1280c2a414adafcfb28137e3d27dca9b83a1f1b7865ca4d7d00dd4cd: Status 404 returned error can't find the container with id 88d245fb1280c2a414adafcfb28137e3d27dca9b83a1f1b7865ca4d7d00dd4cd
Jul 30 03:34:54.834669 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.834646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p2q9q"
Jul 30 03:34:54.842679 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.842654    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519e4a13_90e3_4d40_a097_cecb5171c54c.slice/crio-849625884f6c83f7a4a4541e5b7a8bd6d0b10e2d31bd4c9906b63b6dbaf368d9 WatchSource:0}: Error finding container 849625884f6c83f7a4a4541e5b7a8bd6d0b10e2d31bd4c9906b63b6dbaf368d9: Status 404 returned error can't find the container with id 849625884f6c83f7a4a4541e5b7a8bd6d0b10e2d31bd4c9906b63b6dbaf368d9
Jul 30 03:34:54.850544 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.850524    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n6q7v"
Jul 30 03:34:54.855089 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:54.855070    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fltkz"
Jul 30 03:34:54.859267 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.859226    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1403882_8905_4bed_b140_58eb42b80794.slice/crio-77489aae243cabea2781033bd0416d725216de8189698d90cf6a101fd6bc47e8 WatchSource:0}: Error finding container 77489aae243cabea2781033bd0416d725216de8189698d90cf6a101fd6bc47e8: Status 404 returned error can't find the container with id 77489aae243cabea2781033bd0416d725216de8189698d90cf6a101fd6bc47e8
Jul 30 03:34:54.864228 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:34:54.864191    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c8e946_a078_4b39_912c_fde7009abb28.slice/crio-6d090565c6f177e6ea76103dc51e5aa64714d6806646464b2da42cc5243f2433 WatchSource:0}: Error finding container 6d090565c6f177e6ea76103dc51e5aa64714d6806646464b2da42cc5243f2433: Status 404 returned error can't find the container with id 6d090565c6f177e6ea76103dc51e5aa64714d6806646464b2da42cc5243f2433
Jul 30 03:34:55.019643 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.019563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:55.019776 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.019704    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:34:55.019776 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.019767    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:56.01975029 +0000 UTC m=+2.974384956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:34:55.121014 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.120979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:55.121174 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.121048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:55.121174 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.121152    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:34:55.121285 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.121227    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:34:55.121285 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.121255    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret podName:239554d7-aed4-4f52-8b1e-fdd8b3758c97 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:56.121213791 +0000 UTC m=+3.075848440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret") pod "global-pull-secret-syncer-xfcs5" (UID: "239554d7-aed4-4f52-8b1e-fdd8b3758c97") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:34:55.121285 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.121267    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:34:55.121285 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.121281    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jcm8b for pod openshift-network-diagnostics/network-check-target-qxxxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:34:55.121487 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.121335    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b podName:6e46faea-9a53-4a19-83cd-c1fbde38fae9 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:56.121317544 +0000 UTC m=+3.075952193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jcm8b" (UniqueName: "kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b") pod "network-check-target-qxxxh" (UID: "6e46faea-9a53-4a19-83cd-c1fbde38fae9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:34:55.213060 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.213027    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:34:55.443722 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.443587    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 03:29:54 +0000 UTC" deadline="2028-04-01 00:42:37.482755241 +0000 UTC"
Jul 30 03:34:55.443722 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.443622    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14661h7m42.039137559s"
Jul 30 03:34:55.574917 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.574884    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:55.575090 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.575041    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:34:55.575159 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.575145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:55.575257 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:55.575222    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:34:55.594417 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.594363    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal" event={"ID":"8d00ccf9cbba644f9ba54a54e77d171e","Type":"ContainerStarted","Data":"d9f2b71b37e6b0780d1f8bca280ad43f16e95923569798b1245700f073134ad0"}
Jul 30 03:34:55.601744 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.601710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal" event={"ID":"4f061476c9cab299179308efd01fc989","Type":"ContainerStarted","Data":"33cf82f3dc4000c6d7cae8092ac0abf44a54ff2f16e8120a2900556aa0c401d6"}
Jul 30 03:34:55.614631 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.614544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p2q9q" event={"ID":"519e4a13-90e3-4d40-a097-cecb5171c54c","Type":"ContainerStarted","Data":"849625884f6c83f7a4a4541e5b7a8bd6d0b10e2d31bd4c9906b63b6dbaf368d9"}
Jul 30 03:34:55.623201 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.623168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc" event={"ID":"704164d4-c34d-4ba7-8080-b59d40739cf9","Type":"ContainerStarted","Data":"88d245fb1280c2a414adafcfb28137e3d27dca9b83a1f1b7865ca4d7d00dd4cd"}
Jul 30 03:34:55.646187 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.646125    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5szt" event={"ID":"1236a4cf-a822-4222-ab7b-ecec2387f240","Type":"ContainerStarted","Data":"165d96b6f4ef3ce33ccf43d304647bd2b2738f0b54831d5b2fba4ba620083f78"}
Jul 30 03:34:55.657794 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.657759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerStarted","Data":"505cddf693632430775ac337e85121ba7c815a9938fb61a4aafc25cfedc4eab9"}
Jul 30 03:34:55.669664 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.669573    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22d4x" event={"ID":"f8708fd8-0e95-41fb-914c-54b8f9828bd4","Type":"ContainerStarted","Data":"40f91fed53de04e9d947ce7b3a0098af56c38e09ed4abe1926f0ae2a3409b189"}
Jul 30 03:34:55.682087 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.682060    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:34:55.686122 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.685961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fltkz" event={"ID":"48c8e946-a078-4b39-912c-fde7009abb28","Type":"ContainerStarted","Data":"6d090565c6f177e6ea76103dc51e5aa64714d6806646464b2da42cc5243f2433"}
Jul 30 03:34:55.698269 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.698124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n6q7v" event={"ID":"b1403882-8905-4bed-b140-58eb42b80794","Type":"ContainerStarted","Data":"77489aae243cabea2781033bd0416d725216de8189698d90cf6a101fd6bc47e8"}
Jul 30 03:34:55.718055 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.717718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"3cbc59f29dfc546b5002667a4475ebd4f0a9cf2a9a861ed0b04dcdbb03eb8fab"}
Jul 30 03:34:55.756944 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.756900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8m5mq" event={"ID":"da4390c0-f2af-453b-8000-3ddfc8ace76e","Type":"ContainerStarted","Data":"e0c980047619975c6e362297b7333eeb3f71112114f018c5ce047fb4c22ceffd"}
Jul 30 03:34:55.952534 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:55.952283    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:34:56.031369 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:56.030767    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:56.031369 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.030941    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:34:56.031369 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.031007    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:58.030988568 +0000 UTC m=+4.985623218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:34:56.133807 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:56.133766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:56.133989 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:56.133838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:56.134049 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.134020    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:34:56.134049 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.134038    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:34:56.134149 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.134055    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jcm8b for pod openshift-network-diagnostics/network-check-target-qxxxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:34:56.134149 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.134119    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b podName:6e46faea-9a53-4a19-83cd-c1fbde38fae9 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:58.13409834 +0000 UTC m=+5.088733002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jcm8b" (UniqueName: "kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b") pod "network-check-target-qxxxh" (UID: "6e46faea-9a53-4a19-83cd-c1fbde38fae9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:34:56.134599 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.134579    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:34:56.134684 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.134643    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret podName:239554d7-aed4-4f52-8b1e-fdd8b3758c97 nodeName:}" failed. No retries permitted until 2026-07-30 03:34:58.134626155 +0000 UTC m=+5.089260812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret") pod "global-pull-secret-syncer-xfcs5" (UID: "239554d7-aed4-4f52-8b1e-fdd8b3758c97") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:34:56.444657 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:56.444542    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 03:29:54 +0000 UTC" deadline="2028-01-27 19:16:44.284500116 +0000 UTC"
Jul 30 03:34:56.444657 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:56.444609    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13119h41m47.839895559s"
Jul 30 03:34:56.573247 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:56.572728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:56.573247 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:56.572865    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:34:57.574330 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:57.574019    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:57.574763 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:57.574426    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:34:57.574822 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:57.574079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:57.575019 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:57.574903    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:34:58.053401 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:58.052781    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:58.053401 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.052962    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:34:58.053401 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.053025    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:02.053006927 +0000 UTC m=+9.007641578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:34:58.153716 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:58.153615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:58.153879 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:58.153753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:58.153935 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.153898    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:34:58.153935 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.153917    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:34:58.153935 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.153932    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jcm8b for pod openshift-network-diagnostics/network-check-target-qxxxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:34:58.154042 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.153990    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b podName:6e46faea-9a53-4a19-83cd-c1fbde38fae9 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:02.153972458 +0000 UTC m=+9.108607111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jcm8b" (UniqueName: "kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b") pod "network-check-target-qxxxh" (UID: "6e46faea-9a53-4a19-83cd-c1fbde38fae9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:34:58.154147 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.154132    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:34:58.154229 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.154175    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret podName:239554d7-aed4-4f52-8b1e-fdd8b3758c97 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:02.154162175 +0000 UTC m=+9.108796828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret") pod "global-pull-secret-syncer-xfcs5" (UID: "239554d7-aed4-4f52-8b1e-fdd8b3758c97") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:34:58.572624 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:58.572593    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:34:58.572795 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:58.572740    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:34:59.573797 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:59.573752    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:34:59.574210 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:59.573884    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:34:59.574349 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:34:59.574329    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:34:59.574468 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:34:59.574443    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:00.571975 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:00.571944    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:00.572141 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:00.572057    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:01.573286 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:01.573251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:01.573286 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:01.573285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:01.573793 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:01.573386    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:01.573793 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:01.573772    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:02.086435 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:02.086396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:02.086674 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.086614    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:35:02.086760 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.086692    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:10.086672775 +0000 UTC m=+17.041307427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:35:02.186975 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:02.186934    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:02.187181 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:02.187035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:02.187181 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.187131    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:35:02.187181 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.187157    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:35:02.187181 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.187173    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jcm8b for pod openshift-network-diagnostics/network-check-target-qxxxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:35:02.187425 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.187252    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b podName:6e46faea-9a53-4a19-83cd-c1fbde38fae9 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:10.187216666 +0000 UTC m=+17.141851358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jcm8b" (UniqueName: "kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b") pod "network-check-target-qxxxh" (UID: "6e46faea-9a53-4a19-83cd-c1fbde38fae9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:35:02.187425 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.187136    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:35:02.187425 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.187339    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret podName:239554d7-aed4-4f52-8b1e-fdd8b3758c97 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:10.187319128 +0000 UTC m=+17.141953791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret") pod "global-pull-secret-syncer-xfcs5" (UID: "239554d7-aed4-4f52-8b1e-fdd8b3758c97") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:35:02.572184 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:02.572090    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:02.572369 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:02.572227    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:03.573353 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:03.572861    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:03.573353 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:03.572974    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:03.573353 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:03.573317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:03.573931 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:03.573420    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:04.572249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:04.572199    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:04.572429 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:04.572361    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:05.572252 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:05.572206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:05.572682 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:05.572206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:05.572682 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:05.572356    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:05.572682 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:05.572400    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:06.572172 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:06.572131    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:06.572374 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:06.572308    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:07.572116 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:07.572042    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:07.572116 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:07.572079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:07.572386 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:07.572183    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:07.572386 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:07.572313    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:08.572248 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:08.572219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:08.572445 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:08.572352    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:09.572358 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:09.572327    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:09.572575 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:09.572327    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:09.573662 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:09.573029    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:09.573662 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:09.573305    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:10.146347 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:10.146300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:10.146556 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.146520    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:35:10.146628 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.146608    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:26.14658814 +0000 UTC m=+33.101222789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:35:10.247328 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:10.247292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:10.247492 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:10.247339    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:10.247492 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.247454    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:35:10.247559 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.247524    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret podName:239554d7-aed4-4f52-8b1e-fdd8b3758c97 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:26.247509301 +0000 UTC m=+33.202143949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret") pod "global-pull-secret-syncer-xfcs5" (UID: "239554d7-aed4-4f52-8b1e-fdd8b3758c97") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:35:10.247559 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.247463    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:35:10.247559 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.247555    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:35:10.247655 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.247567    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jcm8b for pod openshift-network-diagnostics/network-check-target-qxxxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:35:10.247655 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.247618    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b podName:6e46faea-9a53-4a19-83cd-c1fbde38fae9 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:26.247604275 +0000 UTC m=+33.202238927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jcm8b" (UniqueName: "kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b") pod "network-check-target-qxxxh" (UID: "6e46faea-9a53-4a19-83cd-c1fbde38fae9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:35:10.572570 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:10.572533    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:10.572987 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:10.572672    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:11.572785 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:11.572751    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:11.573268 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:11.572900    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:11.573268 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:11.572949    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:11.573268 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:11.573063    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:12.572192 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:12.572150    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:12.572405 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:12.572311    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:13.572738 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:13.572698    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:13.573219 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:13.572810    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:13.573219 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:13.572895    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:13.573219 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:13.572985    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:14.573757 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.573301    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:14.573757 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:14.573436    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:14.823858 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.823416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5szt" event={"ID":"1236a4cf-a822-4222-ab7b-ecec2387f240","Type":"ContainerStarted","Data":"cfa2b0a26800efd6e61c7003f6197bd02c51451de89db2fad8c08b2c23f3a081"}
Jul 30 03:35:14.825790 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.825633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerStarted","Data":"865daf580ee33958158076a7b3af3343d94488894ff6fde3689599267aa34bfe"}
Jul 30 03:35:14.829358 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.829036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22d4x" event={"ID":"f8708fd8-0e95-41fb-914c-54b8f9828bd4","Type":"ContainerStarted","Data":"ac462f124256df0c92c696a7255fa3c9847d7445cfc201ad0d69ecbc71e752b4"}
Jul 30 03:35:14.830981 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.830925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fltkz" event={"ID":"48c8e946-a078-4b39-912c-fde7009abb28","Type":"ContainerStarted","Data":"15883ead594aab347adeba0f420328bce25e6952d5b0f1fe17ec5cd3d1ca51af"}
Jul 30 03:35:14.836704 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.836472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"5ffb8300d18a25adce6471f37b2d637edcdd2c2ce413dff20aa5d246b44cde85"}
Jul 30 03:35:14.836810 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.836772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"c7977f9849298da0799a2f645f1e6f8e5fe68936f96738d7f3961835afc77000"}
Jul 30 03:35:14.837955 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.837833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8m5mq" event={"ID":"da4390c0-f2af-453b-8000-3ddfc8ace76e","Type":"ContainerStarted","Data":"1ad44ee9b4fccf557e4d36b79adad4d32967e1bdad31b36674e1b5946a3f8737"}
Jul 30 03:35:14.840220 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.840193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal" event={"ID":"8d00ccf9cbba644f9ba54a54e77d171e","Type":"ContainerStarted","Data":"f227a8a1386f58968ac79dad4577e81d5656e87e85871d18218e4d078bf894d1"}
Jul 30 03:35:14.847501 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.846963    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal" event={"ID":"4f061476c9cab299179308efd01fc989","Type":"ContainerStarted","Data":"a3dfb8bd996cd7d63e304351b469a6438987b79468df5fc8824b2e763ead45ca"}
Jul 30 03:35:14.851803 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.851767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc" event={"ID":"704164d4-c34d-4ba7-8080-b59d40739cf9","Type":"ContainerStarted","Data":"c0a2994a188386df117062cfa1cd52f0539c7152c970fbad6a0b480b49eb2cbb"}
Jul 30 03:35:14.857503 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.857444    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fltkz" podStartSLOduration=2.361386012 podStartE2EDuration="21.857425205s" totalImagesPullingTime="19.496039193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:14.857122321 +0000 UTC m=+21.811756992" watchObservedRunningTime="2026-07-30 03:35:14.857425205 +0000 UTC m=+21.812059889"
Jul 30 03:35:14.857933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.857896    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r5szt" podStartSLOduration=6.23918838 podStartE2EDuration="21.857882874s" totalImagesPullingTime="15.618694494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:14.843055056 +0000 UTC m=+21.797689728" watchObservedRunningTime="2026-07-30 03:35:14.857882874 +0000 UTC m=+21.812517546"
Jul 30 03:35:14.875201 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.875160    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-22d4x" podStartSLOduration=2.234686602 podStartE2EDuration="21.875146296s" totalImagesPullingTime="19.640459694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:14.874912231 +0000 UTC m=+21.829546904" watchObservedRunningTime="2026-07-30 03:35:14.875146296 +0000 UTC m=+21.829780966"
Jul 30 03:35:14.887174 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.887130    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-125.ec2.internal" podStartSLOduration=20.88711394 podStartE2EDuration="20.88711394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:14.886591599 +0000 UTC m=+21.841226271" watchObservedRunningTime="2026-07-30 03:35:14.88711394 +0000 UTC m=+21.841748612"
Jul 30 03:35:14.903684 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:14.903639    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8m5mq" podStartSLOduration=2.268324987 podStartE2EDuration="21.90362509s" totalImagesPullingTime="19.635300103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:14.903316685 +0000 UTC m=+21.857951356" watchObservedRunningTime="2026-07-30 03:35:14.90362509 +0000 UTC m=+21.858259796"
Jul 30 03:35:15.572507 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.572476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:15.572688 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.572484    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:15.572688 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:15.572605    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:15.572688 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:15.572670    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:15.855065 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.854987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n6q7v" event={"ID":"b1403882-8905-4bed-b140-58eb42b80794","Type":"ContainerStarted","Data":"3babf1e130da2083543a09f1af2674ceaa0e558b045651a9032e94690b7ae342"}
Jul 30 03:35:15.858041 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.858016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"49c1911944455f4eb42463f7beb271b74eee5e63907be322677f5a9e4fe59871"}
Jul 30 03:35:15.858168 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.858095    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"9d094919882a5cf45527ac9a162c38a82c428035bc4cdb4e41badc96825bc16a"}
Jul 30 03:35:15.858168 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.858142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"59e33d501ffb196197a6d82fac6b6437913d06b13e5e2d428f01840dd1802074"}
Jul 30 03:35:15.858362 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.858187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"7f9fdceafbd6f94d0e9897b771d7b3d740c960dfb54d2b4c38470f64096baccf"}
Jul 30 03:35:15.859557 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.859536    2560 generic.go:356] "Generic (PLEG): container finished" podID="4f061476c9cab299179308efd01fc989" containerID="a3dfb8bd996cd7d63e304351b469a6438987b79468df5fc8824b2e763ead45ca" exitCode=0
Jul 30 03:35:15.859654 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.859583    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal" event={"ID":"4f061476c9cab299179308efd01fc989","Type":"ContainerDied","Data":"a3dfb8bd996cd7d63e304351b469a6438987b79468df5fc8824b2e763ead45ca"}
Jul 30 03:35:15.860981 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.860959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p2q9q" event={"ID":"519e4a13-90e3-4d40-a097-cecb5171c54c","Type":"ContainerStarted","Data":"c8a79cd6a923e0c86d47ca546f348349d5405a961bdb9f6602aa9a517737bfd0"}
Jul 30 03:35:15.862323 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.862302    2560 generic.go:356] "Generic (PLEG): container finished" podID="30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd" containerID="865daf580ee33958158076a7b3af3343d94488894ff6fde3689599267aa34bfe" exitCode=0
Jul 30 03:35:15.862403 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.862371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerDied","Data":"865daf580ee33958158076a7b3af3343d94488894ff6fde3689599267aa34bfe"}
Jul 30 03:35:15.867940 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.867905    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n6q7v" podStartSLOduration=3.367286717 podStartE2EDuration="22.867894634s" totalImagesPullingTime="19.500607917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:15.86775145 +0000 UTC m=+22.822386121" watchObservedRunningTime="2026-07-30 03:35:15.867894634 +0000 UTC m=+22.822529305"
Jul 30 03:35:15.899356 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.899301    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p2q9q" podStartSLOduration=11.313716116 podStartE2EDuration="22.899281568s" totalImagesPullingTime="11.585565452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:15.899011711 +0000 UTC m=+22.853646383" watchObservedRunningTime="2026-07-30 03:35:15.899281568 +0000 UTC m=+22.853916240"
Jul 30 03:35:15.976613 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:15.976466    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 03:35:16.483352 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:16.483209    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T03:35:15.976484165Z","UUID":"9166ded6-195f-4b4d-a71a-6b52272b9dd3","Handler":null,"Name":"","Endpoint":""}
Jul 30 03:35:16.485411 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:16.485387    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 03:35:16.485546 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:16.485434    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 03:35:16.572357 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:16.572320    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:16.572539 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:16.572486    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:16.866766 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:16.866688    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal" event={"ID":"4f061476c9cab299179308efd01fc989","Type":"ContainerStarted","Data":"4c1c595188a8008ba5c095afeb78f692cac40fae9f0c989d8ccba2b0bca225ba"}
Jul 30 03:35:16.868761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:16.868725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc" event={"ID":"704164d4-c34d-4ba7-8080-b59d40739cf9","Type":"ContainerStarted","Data":"f252a9aaa05e3544e0126ab7174a9f24314e728c678f4abe674c96092a9b7180"}
Jul 30 03:35:17.326104 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:17.326070    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:35:17.326801 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:17.326781    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:35:17.339983 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:17.339932    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-125.ec2.internal" podStartSLOduration=23.339915997 podStartE2EDuration="23.339915997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:16.882085549 +0000 UTC m=+23.836720222" watchObservedRunningTime="2026-07-30 03:35:17.339915997 +0000 UTC m=+24.294550667"
Jul 30 03:35:17.572829 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:17.572792    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:17.573019 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:17.572806    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:17.573019 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:17.572936    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:17.573154 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:17.573014    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:17.873271 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:17.873219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc" event={"ID":"704164d4-c34d-4ba7-8080-b59d40739cf9","Type":"ContainerStarted","Data":"5dd6bb464c1c4bc01b7cbc97a2b84244c1d03e7715e4f3eabd1a1e0c5e812acf"}
Jul 30 03:35:17.879382 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:17.878284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"c1b9e4138cfc32c2d65bd995029848d8167843ac0aaf765633c62eac9489531e"}
Jul 30 03:35:17.888910 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:17.888868    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n82pc" podStartSLOduration=3.238374959 podStartE2EDuration="24.888851148s" totalImagesPullingTime="21.650476189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:17.888794665 +0000 UTC m=+24.843429343" watchObservedRunningTime="2026-07-30 03:35:17.888851148 +0000 UTC m=+24.843485820"
Jul 30 03:35:18.572682 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:18.572651    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:18.572878 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:18.572787    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:19.572948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:19.572748    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:19.573397 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:19.572803    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:19.573397 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:19.573027    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:19.573397 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:19.573138    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:19.878066 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:19.878037    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:35:19.878193 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:19.878172    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 30 03:35:19.878777 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:19.878758    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r5szt"
Jul 30 03:35:20.572168 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:20.572135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:20.572333 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:20.572280    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:20.887808 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:20.887775    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" event={"ID":"ebf7b93f-53c0-4c89-aa84-d9188b09e918","Type":"ContainerStarted","Data":"2d858644ee75bcb089987c9cd4df056f4b82248d65f671331a363c3e63c6aab4"}
Jul 30 03:35:20.888706 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:20.888034    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:35:20.889655 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:20.889631    2560 generic.go:356] "Generic (PLEG): container finished" podID="30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd" containerID="83aa6d563f8cb2941be8b5bfc2e866fcb520e13fda053c4632040faecaf39228" exitCode=0
Jul 30 03:35:20.889921 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:20.889659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerDied","Data":"83aa6d563f8cb2941be8b5bfc2e866fcb520e13fda053c4632040faecaf39228"}
Jul 30 03:35:20.904562 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:20.904538    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:35:20.915804 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:20.915762    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" podStartSLOduration=8.086564338 podStartE2EDuration="27.915748553s" totalImagesPullingTime="19.829184215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:20.915394287 +0000 UTC m=+27.870028957" watchObservedRunningTime="2026-07-30 03:35:20.915748553 +0000 UTC m=+27.870383270"
Jul 30 03:35:21.572990 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:21.572740    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:21.573195 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:21.572807    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:21.573195 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:21.573099    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:21.573195 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:21.573174    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:21.894751 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:21.894718    2560 generic.go:356] "Generic (PLEG): container finished" podID="30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd" containerID="617f00c27c6b021c2ad9e725e88570ab9f38cd74f96239af0f5dd54e200c3915" exitCode=0
Jul 30 03:35:21.895197 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:21.894819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerDied","Data":"617f00c27c6b021c2ad9e725e88570ab9f38cd74f96239af0f5dd54e200c3915"}
Jul 30 03:35:21.895197 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:21.894905    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 30 03:35:21.895469 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:21.895450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:35:21.913393 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:21.913366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:35:22.159954 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:22.159922    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qxxxh"]
Jul 30 03:35:22.160147 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:22.160043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:22.160270 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:22.160154    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:22.163195 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:22.163171    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pc47f"]
Jul 30 03:35:22.163313 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:22.163306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:22.163456 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:22.163426    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:22.163718 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:22.163697    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xfcs5"]
Jul 30 03:35:22.163784 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:22.163774    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:22.163874 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:22.163857    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:22.896916 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:22.896885    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 30 03:35:23.573374 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:23.573338    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:23.573554 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:23.573453    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:23.573554 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:23.573511    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:23.573674 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:23.573650    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:23.901358 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:23.901319    2560 generic.go:356] "Generic (PLEG): container finished" podID="30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd" containerID="5559b074aedb9b7157622b1a1adb4f6a64c77ffc657acd2ae941543dec16099c" exitCode=0
Jul 30 03:35:23.901810 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:23.901357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerDied","Data":"5559b074aedb9b7157622b1a1adb4f6a64c77ffc657acd2ae941543dec16099c"}
Jul 30 03:35:23.901810 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:23.901517    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 30 03:35:23.982340 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:23.982305    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:35:24.573159 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:24.572952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:24.573340 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:24.573286    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:24.921903 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:24.921842    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4" podUID="ebf7b93f-53c0-4c89-aa84-d9188b09e918" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 30 03:35:25.572458 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:25.572399    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:25.572649 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:25.572404    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:25.572649 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:25.572522    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:25.572649 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:25.572593    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:26.166507 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:26.166477    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:26.166906 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.166603    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:35:26.166906 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.166677    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:58.166656746 +0000 UTC m=+65.121291396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:35:26.267453 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:26.267373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:26.267453 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:26.267429    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:26.267672 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.267536    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:35:26.267672 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.267559    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:35:26.267672 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.267576    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:35:26.267672 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.267590    2560 projected.go:196] Error preparing data for projected volume kube-api-access-jcm8b for pod openshift-network-diagnostics/network-check-target-qxxxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:35:26.267672 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.267611    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret podName:239554d7-aed4-4f52-8b1e-fdd8b3758c97 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:58.267588958 +0000 UTC m=+65.222223709 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret") pod "global-pull-secret-syncer-xfcs5" (UID: "239554d7-aed4-4f52-8b1e-fdd8b3758c97") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:35:26.267672 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.267635    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b podName:6e46faea-9a53-4a19-83cd-c1fbde38fae9 nodeName:}" failed. No retries permitted until 2026-07-30 03:35:58.267622572 +0000 UTC m=+65.222257221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jcm8b" (UniqueName: "kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b") pod "network-check-target-qxxxh" (UID: "6e46faea-9a53-4a19-83cd-c1fbde38fae9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:35:26.572013 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:26.571929    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:26.572167 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:26.572064    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfcs5" podUID="239554d7-aed4-4f52-8b1e-fdd8b3758c97"
Jul 30 03:35:27.572316 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:27.572288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:27.572316 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:27.572321    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:27.572806 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:27.572399    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxxxh" podUID="6e46faea-9a53-4a19-83cd-c1fbde38fae9"
Jul 30 03:35:27.572806 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:27.572520    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:35:27.913843 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:27.913810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerStarted","Data":"b366a3b3638a3392d9af03f357bde96907e3d477204acd3d62ea4bcba5aad43d"}
Jul 30 03:35:28.371911 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.371885    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-125.ec2.internal" event="NodeReady"
Jul 30 03:35:28.372065 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.372016    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 03:35:28.419219 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.419185    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ctdpt"]
Jul 30 03:35:28.422245 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.422214    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ctdpt"
Jul 30 03:35:28.424073 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.424044    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 03:35:28.424073 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.424050    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 03:35:28.424264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.424155    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 03:35:28.424264 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.424201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pbdqz\""
Jul 30 03:35:28.426647 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.426627    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7t5ct"]
Jul 30 03:35:28.430083 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.430066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.431665 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.431651    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 03:35:28.432029 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.432017    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nmkwv\""
Jul 30 03:35:28.432269 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.432251    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 03:35:28.433816 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.433792    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ctdpt"]
Jul 30 03:35:28.439753 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.439734    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7t5ct"]
Jul 30 03:35:28.572817 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.572774    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:28.577462 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.577434    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 03:35:28.585537 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.585516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqb5\" (UniqueName: \"kubernetes.io/projected/58a0e78a-b220-4eb2-8435-7298c7e001f4-kube-api-access-7kqb5\") pod \"ingress-canary-ctdpt\" (UID: \"58a0e78a-b220-4eb2-8435-7298c7e001f4\") " pod="openshift-ingress-canary/ingress-canary-ctdpt"
Jul 30 03:35:28.585683 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.585559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-config-volume\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.585683 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.585576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-tmp-dir\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.585683 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.585654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.585683 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.585679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58a0e78a-b220-4eb2-8435-7298c7e001f4-cert\") pod \"ingress-canary-ctdpt\" (UID: \"58a0e78a-b220-4eb2-8435-7298c7e001f4\") " pod="openshift-ingress-canary/ingress-canary-ctdpt"
Jul 30 03:35:28.585810 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.585713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngtr\" (UniqueName: \"kubernetes.io/projected/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-kube-api-access-vngtr\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.686880 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.686848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.687054 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.686893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58a0e78a-b220-4eb2-8435-7298c7e001f4-cert\") pod \"ingress-canary-ctdpt\" (UID: \"58a0e78a-b220-4eb2-8435-7298c7e001f4\") " pod="openshift-ingress-canary/ingress-canary-ctdpt"
Jul 30 03:35:28.687054 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.686935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngtr\" (UniqueName: \"kubernetes.io/projected/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-kube-api-access-vngtr\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.687054 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.686961    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqb5\" (UniqueName: \"kubernetes.io/projected/58a0e78a-b220-4eb2-8435-7298c7e001f4-kube-api-access-7kqb5\") pod \"ingress-canary-ctdpt\" (UID: \"58a0e78a-b220-4eb2-8435-7298c7e001f4\") " pod="openshift-ingress-canary/ingress-canary-ctdpt"
Jul 30 03:35:28.687054 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:28.686993    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:35:28.687054 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.687000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-config-volume\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.687054 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:28.687055    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:35:29.187033835 +0000 UTC m=+36.141668486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:35:28.687365 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.687082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-tmp-dir\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.687432 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.687415    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-tmp-dir\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.687654 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.687604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-config-volume\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.691284 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.691261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58a0e78a-b220-4eb2-8435-7298c7e001f4-cert\") pod \"ingress-canary-ctdpt\" (UID: \"58a0e78a-b220-4eb2-8435-7298c7e001f4\") " pod="openshift-ingress-canary/ingress-canary-ctdpt"
Jul 30 03:35:28.695076 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.695053    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngtr\" (UniqueName: \"kubernetes.io/projected/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-kube-api-access-vngtr\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:28.695942 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.695920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqb5\" (UniqueName: \"kubernetes.io/projected/58a0e78a-b220-4eb2-8435-7298c7e001f4-kube-api-access-7kqb5\") pod \"ingress-canary-ctdpt\" (UID: \"58a0e78a-b220-4eb2-8435-7298c7e001f4\") " pod="openshift-ingress-canary/ingress-canary-ctdpt"
Jul 30 03:35:28.731724 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.731691    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ctdpt"
Jul 30 03:35:28.883186 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.883156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ctdpt"]
Jul 30 03:35:28.888114 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:35:28.887977    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a0e78a_b220_4eb2_8435_7298c7e001f4.slice/crio-0c3d0e3d87a0b2090ba02a2e29dd1c35cf4bb6434a06be2e8d243248e1b10bef WatchSource:0}: Error finding container 0c3d0e3d87a0b2090ba02a2e29dd1c35cf4bb6434a06be2e8d243248e1b10bef: Status 404 returned error can't find the container with id 0c3d0e3d87a0b2090ba02a2e29dd1c35cf4bb6434a06be2e8d243248e1b10bef
Jul 30 03:35:28.919051 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.919021    2560 generic.go:356] "Generic (PLEG): container finished" podID="30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd" containerID="b366a3b3638a3392d9af03f357bde96907e3d477204acd3d62ea4bcba5aad43d" exitCode=0
Jul 30 03:35:28.919183 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.919089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerDied","Data":"b366a3b3638a3392d9af03f357bde96907e3d477204acd3d62ea4bcba5aad43d"}
Jul 30 03:35:28.920202 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:28.920181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ctdpt" event={"ID":"58a0e78a-b220-4eb2-8435-7298c7e001f4","Type":"ContainerStarted","Data":"0c3d0e3d87a0b2090ba02a2e29dd1c35cf4bb6434a06be2e8d243248e1b10bef"}
Jul 30 03:35:29.192331 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.192297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:29.192489 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:29.192418    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:35:29.192489 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:29.192479    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:35:30.19246015 +0000 UTC m=+37.147094815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:35:29.572877 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.572778    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:29.572877 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.572833    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:29.575353 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.575330    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 03:35:29.575476 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.575359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 03:35:29.575635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.575616    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jkrkk\""
Jul 30 03:35:29.575719 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.575661    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bvtgp\""
Jul 30 03:35:29.576007 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.575980    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 03:35:29.927050 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.927011    2560 generic.go:356] "Generic (PLEG): container finished" podID="30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd" containerID="519cc90bf199b663e832f35293ba2552338f25999cafbfef1f65e657f5163006" exitCode=0
Jul 30 03:35:29.927289 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:29.927065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerDied","Data":"519cc90bf199b663e832f35293ba2552338f25999cafbfef1f65e657f5163006"}
Jul 30 03:35:30.200044 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:30.199965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:30.200191 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:30.200082    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:35:30.200191 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:30.200157    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:35:32.200140042 +0000 UTC m=+39.154774715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:35:30.933289 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:30.933249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" event={"ID":"30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd","Type":"ContainerStarted","Data":"54f525f6b75eff563a3db1c1c9ef58176a1643198cbc300c358a8666ec6218c6"}
Jul 30 03:35:30.954017 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:30.953956    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mlpxh" podStartSLOduration=8.665196779 podStartE2EDuration="37.953937769s" totalImagesPullingTime="29.28874099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:30.952821059 +0000 UTC m=+37.907455771" watchObservedRunningTime="2026-07-30 03:35:30.953937769 +0000 UTC m=+37.908572440"
Jul 30 03:35:31.937076 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:31.937030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ctdpt" event={"ID":"58a0e78a-b220-4eb2-8435-7298c7e001f4","Type":"ContainerStarted","Data":"62df34253566ae87edad08468fd529b3410455337128334bbc25c8fa075b2148"}
Jul 30 03:35:31.951117 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:31.951074    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ctdpt" podStartSLOduration=1.392875364 podStartE2EDuration="3.951058514s" totalImagesPullingTime="2.55818315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:35:31.95090605 +0000 UTC m=+38.905540725" watchObservedRunningTime="2026-07-30 03:35:31.951058514 +0000 UTC m=+38.905693162"
Jul 30 03:35:32.214117 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:32.214029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:32.214291 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:32.214180    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:35:32.214291 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:32.214278    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:35:36.214262227 +0000 UTC m=+43.168896876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:35:36.241484 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:36.241436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:36.241851 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:36.241595    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:35:36.241851 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:36.241656    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:35:44.241638842 +0000 UTC m=+51.196273494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:35:44.297030 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:44.296988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:35:44.297641 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:44.297175    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:35:44.297641 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:44.297280    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:36:00.297257271 +0000 UTC m=+67.251891922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:35:54.913406 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:54.913377    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nclz4"
Jul 30 03:35:58.191626 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.191572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:35:58.193783 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.193764    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 03:35:58.202289 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:58.202269    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 30 03:35:58.202359 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:35:58.202328    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:37:02.202311085 +0000 UTC m=+129.156945735 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : secret "metrics-daemon-secret" not found
Jul 30 03:35:58.292025 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.291989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:58.292142 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.292067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:58.294104 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.294084    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 03:35:58.294152 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.294084    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 03:35:58.303615 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.303596    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 03:35:58.306202 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.306181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/239554d7-aed4-4f52-8b1e-fdd8b3758c97-original-pull-secret\") pod \"global-pull-secret-syncer-xfcs5\" (UID: \"239554d7-aed4-4f52-8b1e-fdd8b3758c97\") " pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:58.315172 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.315149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcm8b\" (UniqueName: \"kubernetes.io/projected/6e46faea-9a53-4a19-83cd-c1fbde38fae9-kube-api-access-jcm8b\") pod \"network-check-target-qxxxh\" (UID: \"6e46faea-9a53-4a19-83cd-c1fbde38fae9\") " pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:58.393275 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.393228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jkrkk\""
Jul 30 03:35:58.401581 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.401564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:35:58.520977 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.520943    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qxxxh"]
Jul 30 03:35:58.524616 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:35:58.524584    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e46faea_9a53_4a19_83cd_c1fbde38fae9.slice/crio-25ac721b5a040f4d0cac55714409804b9d396f8dabb97c41a3f7d58479c5e137 WatchSource:0}: Error finding container 25ac721b5a040f4d0cac55714409804b9d396f8dabb97c41a3f7d58479c5e137: Status 404 returned error can't find the container with id 25ac721b5a040f4d0cac55714409804b9d396f8dabb97c41a3f7d58479c5e137
Jul 30 03:35:58.580983 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.580951    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfcs5"
Jul 30 03:35:58.697401 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.697367    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xfcs5"]
Jul 30 03:35:58.700959 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:35:58.700903    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239554d7_aed4_4f52_8b1e_fdd8b3758c97.slice/crio-4ecc7cae0227ad60c38bc3562965be5c82327659364197a7cf4ef676939e1fcb WatchSource:0}: Error finding container 4ecc7cae0227ad60c38bc3562965be5c82327659364197a7cf4ef676939e1fcb: Status 404 returned error can't find the container with id 4ecc7cae0227ad60c38bc3562965be5c82327659364197a7cf4ef676939e1fcb
Jul 30 03:35:58.997221 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.997121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xfcs5" event={"ID":"239554d7-aed4-4f52-8b1e-fdd8b3758c97","Type":"ContainerStarted","Data":"4ecc7cae0227ad60c38bc3562965be5c82327659364197a7cf4ef676939e1fcb"}
Jul 30 03:35:58.998450 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:35:58.998421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qxxxh" event={"ID":"6e46faea-9a53-4a19-83cd-c1fbde38fae9","Type":"ContainerStarted","Data":"25ac721b5a040f4d0cac55714409804b9d396f8dabb97c41a3f7d58479c5e137"}
Jul 30 03:36:00.304061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:00.303979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:36:00.304541 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:36:00.304150    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:36:00.304541 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:36:00.304259    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:36:32.304220843 +0000 UTC m=+99.258855506 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:36:04.012670 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:04.012636    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xfcs5" event={"ID":"239554d7-aed4-4f52-8b1e-fdd8b3758c97","Type":"ContainerStarted","Data":"09c9e97ab3c66f82f74c6b456d383ff74ff593a04b8460cf76ed7627cdbb30ae"}
Jul 30 03:36:04.014088 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:04.014063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qxxxh" event={"ID":"6e46faea-9a53-4a19-83cd-c1fbde38fae9","Type":"ContainerStarted","Data":"874dab98851086010e35f15d5d68d2862dbb88a419deedee3b7a6fb3cf09ba29"}
Jul 30 03:36:04.014200 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:04.014110    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:36:04.027160 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:04.027108    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xfcs5" podStartSLOduration=66.546766018 podStartE2EDuration="1m11.027089866s" totalImagesPullingTime="4.480323848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:36:04.026449482 +0000 UTC m=+70.981084153" watchObservedRunningTime="2026-07-30 03:36:04.027089866 +0000 UTC m=+70.981724537"
Jul 30 03:36:04.039026 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:04.038981    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qxxxh" podStartSLOduration=66.387228676 podStartE2EDuration="1m11.038966846s" totalImagesPullingTime="4.65173817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:36:04.038283221 +0000 UTC m=+70.992917889" watchObservedRunningTime="2026-07-30 03:36:04.038966846 +0000 UTC m=+70.993601516"
Jul 30 03:36:23.152532 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.152401    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"]
Jul 30 03:36:23.156514 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.156498    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.158370 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.158350    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 30 03:36:23.158938 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.158916    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 30 03:36:23.159031 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.158962    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 30 03:36:23.159474 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.159457    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 30 03:36:23.165851 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.165827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"]
Jul 30 03:36:23.259181 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.259139    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/175252c5-8638-4e52-bc51-a008e12deb51-tmp\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.259374 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.259212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mchg\" (UniqueName: \"kubernetes.io/projected/175252c5-8638-4e52-bc51-a008e12deb51-kube-api-access-9mchg\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.259374 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.259252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/175252c5-8638-4e52-bc51-a008e12deb51-klusterlet-config\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.360153 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.360108    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/175252c5-8638-4e52-bc51-a008e12deb51-tmp\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.360290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.360174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mchg\" (UniqueName: \"kubernetes.io/projected/175252c5-8638-4e52-bc51-a008e12deb51-kube-api-access-9mchg\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.360290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.360196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/175252c5-8638-4e52-bc51-a008e12deb51-klusterlet-config\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.360513 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.360494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/175252c5-8638-4e52-bc51-a008e12deb51-tmp\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.362696 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.362675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/175252c5-8638-4e52-bc51-a008e12deb51-klusterlet-config\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.368560 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.368532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mchg\" (UniqueName: \"kubernetes.io/projected/175252c5-8638-4e52-bc51-a008e12deb51-kube-api-access-9mchg\") pod \"klusterlet-addon-workmgr-674767cc4f-kr8jp\" (UID: \"175252c5-8638-4e52-bc51-a008e12deb51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.467439 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.467349    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:23.589345 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:23.589264    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"]
Jul 30 03:36:23.593493 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:36:23.593462    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175252c5_8638_4e52_bc51_a008e12deb51.slice/crio-277b1c0e955dff202819852b82afa46d2bac94e098a71d825c9640994646fcdb WatchSource:0}: Error finding container 277b1c0e955dff202819852b82afa46d2bac94e098a71d825c9640994646fcdb: Status 404 returned error can't find the container with id 277b1c0e955dff202819852b82afa46d2bac94e098a71d825c9640994646fcdb
Jul 30 03:36:24.062991 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:24.062948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp" event={"ID":"175252c5-8638-4e52-bc51-a008e12deb51","Type":"ContainerStarted","Data":"277b1c0e955dff202819852b82afa46d2bac94e098a71d825c9640994646fcdb"}
Jul 30 03:36:32.320698 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:32.320613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:36:32.321145 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:36:32.320797    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:36:32.321145 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:36:32.320888    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:37:36.320864473 +0000 UTC m=+163.275499123 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:36:35.018630 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:35.018596    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qxxxh"
Jul 30 03:36:35.092216 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:35.092184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp" event={"ID":"175252c5-8638-4e52-bc51-a008e12deb51","Type":"ContainerStarted","Data":"aefb9e4d00965332ebfea24b6610c0804407fbba1875e7c805ffb46af309ec24"}
Jul 30 03:36:35.092440 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:35.092423    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:35.093932 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:35.093908    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:36:35.122071 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:36:35.122012    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp" podStartSLOduration=1.528596382 podStartE2EDuration="12.121993614s" totalImagesPullingTime="10.593397232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:36:35.108783177 +0000 UTC m=+102.063417844" watchObservedRunningTime="2026-07-30 03:36:35.121993614 +0000 UTC m=+102.076628286"
Jul 30 03:37:02.223209 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:37:02.223168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:37:02.223730 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:37:02.223348    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 30 03:37:02.223730 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:37:02.223440    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:39:04.223420555 +0000 UTC m=+251.178055206 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : secret "metrics-daemon-secret" not found
Jul 30 03:37:31.438755 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:37:31.438693    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7t5ct" podUID="d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa"
Jul 30 03:37:32.228933 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:37:32.228902    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:37:32.584888 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:37:32.584799    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:37:36.358855 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:37:36.358814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:37:36.359228 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:37:36.358935    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:37:36.359228 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:37:36.358986    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls podName:d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa nodeName:}" failed. No retries permitted until 2026-07-30 03:39:38.358972126 +0000 UTC m=+285.313606775 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls") pod "dns-default-7t5ct" (UID: "d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa") : secret "dns-default-metrics-tls" not found
Jul 30 03:37:46.572169 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:37:46.572069    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:38:34.381936 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:34.381900    2560 generic.go:356] "Generic (PLEG): container finished" podID="175252c5-8638-4e52-bc51-a008e12deb51" containerID="aefb9e4d00965332ebfea24b6610c0804407fbba1875e7c805ffb46af309ec24" exitCode=1
Jul 30 03:38:34.382385 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:34.381957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp" event={"ID":"175252c5-8638-4e52-bc51-a008e12deb51","Type":"ContainerDied","Data":"aefb9e4d00965332ebfea24b6610c0804407fbba1875e7c805ffb46af309ec24"}
Jul 30 03:38:34.382385 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:34.382288    2560 scope.go:117] "RemoveContainer" containerID="aefb9e4d00965332ebfea24b6610c0804407fbba1875e7c805ffb46af309ec24"
Jul 30 03:38:35.092694 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:35.092610    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:38:35.386161 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:35.386138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp" event={"ID":"175252c5-8638-4e52-bc51-a008e12deb51","Type":"ContainerStarted","Data":"bd9f3c082d198d0800f2b32a101a32a743f6cfaa298bbde6b4dbd770def3466a"}
Jul 30 03:38:35.386563 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:35.386347    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:38:35.386939 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:35.386921    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674767cc4f-kr8jp"
Jul 30 03:38:52.059743 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.059559    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn"]
Jul 30 03:38:52.062466 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.062442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn"
Jul 30 03:38:52.064340 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.064321    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 30 03:38:52.064820 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.064803    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gs6b8\""
Jul 30 03:38:52.064926 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.064837    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:38:52.073801 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.073781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn"]
Jul 30 03:38:52.156464 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.156415    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"]
Jul 30 03:38:52.159149 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.159130    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"]
Jul 30 03:38:52.159282 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.159262    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.160915 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.160890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drhnp\" (UniqueName: \"kubernetes.io/projected/a1e976d1-c889-442b-834d-fd69e9bca652-kube-api-access-drhnp\") pod \"volume-data-source-validator-549c6d54cf-vx4rn\" (UID: \"a1e976d1-c889-442b-834d-fd69e9bca652\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn"
Jul 30 03:38:52.161513 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.161490    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 30 03:38:52.161635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.161623    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 30 03:38:52.161697 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.161651    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:38:52.161746 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.161710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m6zbv\""
Jul 30 03:38:52.161967 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.161943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:52.162075 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.162041    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 30 03:38:52.162507 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.162476    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65954c5f6d-hfc6f"]
Jul 30 03:38:52.165203 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.165184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.166096 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.166073    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j56jf\""
Jul 30 03:38:52.166194 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.166111    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 30 03:38:52.167219 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.167044    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:38:52.168113 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.168097    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vqpns\""
Jul 30 03:38:52.168616 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.168601    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 30 03:38:52.168992 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.168976    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 30 03:38:52.169503 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.169483    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 30 03:38:52.175777 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.175758    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 30 03:38:52.179836 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.179813    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"]
Jul 30 03:38:52.181009 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.180988    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"]
Jul 30 03:38:52.181572 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.181551    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 30 03:38:52.202581 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.202553    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65954c5f6d-hfc6f"]
Jul 30 03:38:52.261995 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.261961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-certificates\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.262196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx4rb\" (UniqueName: \"kubernetes.io/projected/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-kube-api-access-zx4rb\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:52.262196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:52.262196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-trusted-ca\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.262196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-installation-pull-secrets\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.262196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drhnp\" (UniqueName: \"kubernetes.io/projected/a1e976d1-c889-442b-834d-fd69e9bca652-kube-api-access-drhnp\") pod \"volume-data-source-validator-549c6d54cf-vx4rn\" (UID: \"a1e976d1-c889-442b-834d-fd69e9bca652\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn"
Jul 30 03:38:52.262196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d54750-5010-4150-8945-09a630b12105-config\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.262196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.262519 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262316    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccqnj\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-kube-api-access-ccqnj\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.262519 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-image-registry-private-configuration\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.262519 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-ca-trust-extracted\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.262519 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262449    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62d54750-5010-4150-8945-09a630b12105-serving-cert\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.262519 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262471    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-bound-sa-token\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.262519 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.262486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxcp\" (UniqueName: \"kubernetes.io/projected/62d54750-5010-4150-8945-09a630b12105-kube-api-access-wgxcp\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.269110 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.269087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drhnp\" (UniqueName: \"kubernetes.io/projected/a1e976d1-c889-442b-834d-fd69e9bca652-kube-api-access-drhnp\") pod \"volume-data-source-validator-549c6d54cf-vx4rn\" (UID: \"a1e976d1-c889-442b-834d-fd69e9bca652\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn"
Jul 30 03:38:52.363772 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.363678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d54750-5010-4150-8945-09a630b12105-config\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.363772 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.363715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.363772 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.363735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccqnj\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-kube-api-access-ccqnj\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.364001 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.363829    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:38:52.364001 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.363842    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65954c5f6d-hfc6f: secret "image-registry-tls" not found
Jul 30 03:38:52.364001 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.363860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-image-registry-private-configuration\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.364001 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.363913    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls podName:a5d33d98-cae0-4ef5-94f2-e133d0a2d51b nodeName:}" failed. No retries permitted until 2026-07-30 03:38:52.863890394 +0000 UTC m=+239.818525065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls") pod "image-registry-65954c5f6d-hfc6f" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b") : secret "image-registry-tls" not found
Jul 30 03:38:52.364001 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.363932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-ca-trust-extracted\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.364001 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.363954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62d54750-5010-4150-8945-09a630b12105-serving-cert\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.364001 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.363979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-bound-sa-token\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.364001 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxcp\" (UniqueName: \"kubernetes.io/projected/62d54750-5010-4150-8945-09a630b12105-kube-api-access-wgxcp\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.364415 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-certificates\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.364415 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364090    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx4rb\" (UniqueName: \"kubernetes.io/projected/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-kube-api-access-zx4rb\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:52.364415 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364143    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:52.364415 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-trusted-ca\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.364415 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-installation-pull-secrets\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.364415 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d54750-5010-4150-8945-09a630b12105-config\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.364415 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-ca-trust-extracted\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.364753 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.364445    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:38:52.364753 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.364506    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls podName:b9fcc12e-b3e4-4064-a53e-3c250308aa2e nodeName:}" failed. No retries permitted until 2026-07-30 03:38:52.864488963 +0000 UTC m=+239.819123635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls") pod "cluster-samples-operator-6748489656-9qds7" (UID: "b9fcc12e-b3e4-4064-a53e-3c250308aa2e") : secret "samples-operator-tls" not found
Jul 30 03:38:52.364881 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.364862    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-certificates\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.365520 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.365497    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-trusted-ca\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.366663 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.366637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-installation-pull-secrets\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.366755 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.366680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62d54750-5010-4150-8945-09a630b12105-serving-cert\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.366811 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.366791    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-image-registry-private-configuration\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.370690 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.370673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn"
Jul 30 03:38:52.374497 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.374474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-bound-sa-token\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.374610 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.374589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx4rb\" (UniqueName: \"kubernetes.io/projected/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-kube-api-access-zx4rb\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:52.374676 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.374642    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxcp\" (UniqueName: \"kubernetes.io/projected/62d54750-5010-4150-8945-09a630b12105-kube-api-access-wgxcp\") pod \"service-ca-operator-5d6c75f84f-j56wx\" (UID: \"62d54750-5010-4150-8945-09a630b12105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.375121 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.374803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccqnj\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-kube-api-access-ccqnj\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.469855 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.469820    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"
Jul 30 03:38:52.494084 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.494052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn"]
Jul 30 03:38:52.497657 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:38:52.497632    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e976d1_c889_442b_834d_fd69e9bca652.slice/crio-48fb280c507f6725dd8b2efb66d495043891ca0fa76e7ba4354b135600838155 WatchSource:0}: Error finding container 48fb280c507f6725dd8b2efb66d495043891ca0fa76e7ba4354b135600838155: Status 404 returned error can't find the container with id 48fb280c507f6725dd8b2efb66d495043891ca0fa76e7ba4354b135600838155
Jul 30 03:38:52.594071 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.594040    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx"]
Jul 30 03:38:52.597135 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:38:52.597104    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d54750_5010_4150_8945_09a630b12105.slice/crio-6ee5d86186fbebbbf5fd1484e326b7171156b71c4ddfdfc978eefcb7b4806bbf WatchSource:0}: Error finding container 6ee5d86186fbebbbf5fd1484e326b7171156b71c4ddfdfc978eefcb7b4806bbf: Status 404 returned error can't find the container with id 6ee5d86186fbebbbf5fd1484e326b7171156b71c4ddfdfc978eefcb7b4806bbf
Jul 30 03:38:52.867058 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.867005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:52.867058 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:52.867069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:52.867336 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.867162    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:38:52.867336 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.867247    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls podName:b9fcc12e-b3e4-4064-a53e-3c250308aa2e nodeName:}" failed. No retries permitted until 2026-07-30 03:38:53.867213256 +0000 UTC m=+240.821847904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls") pod "cluster-samples-operator-6748489656-9qds7" (UID: "b9fcc12e-b3e4-4064-a53e-3c250308aa2e") : secret "samples-operator-tls" not found
Jul 30 03:38:52.867336 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.867170    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:38:52.867336 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.867269    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65954c5f6d-hfc6f: secret "image-registry-tls" not found
Jul 30 03:38:52.867336 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:52.867312    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls podName:a5d33d98-cae0-4ef5-94f2-e133d0a2d51b nodeName:}" failed. No retries permitted until 2026-07-30 03:38:53.867300092 +0000 UTC m=+240.821934741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls") pod "image-registry-65954c5f6d-hfc6f" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b") : secret "image-registry-tls" not found
Jul 30 03:38:53.433992 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:53.433951    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx" event={"ID":"62d54750-5010-4150-8945-09a630b12105","Type":"ContainerStarted","Data":"6ee5d86186fbebbbf5fd1484e326b7171156b71c4ddfdfc978eefcb7b4806bbf"}
Jul 30 03:38:53.435445 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:53.435419    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn" event={"ID":"a1e976d1-c889-442b-834d-fd69e9bca652","Type":"ContainerStarted","Data":"48fb280c507f6725dd8b2efb66d495043891ca0fa76e7ba4354b135600838155"}
Jul 30 03:38:53.876225 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:53.876185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:53.876423 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:53.876268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:53.876423 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:53.876365    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:38:53.876510 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:53.876436    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls podName:b9fcc12e-b3e4-4064-a53e-3c250308aa2e nodeName:}" failed. No retries permitted until 2026-07-30 03:38:55.876419029 +0000 UTC m=+242.831053699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls") pod "cluster-samples-operator-6748489656-9qds7" (UID: "b9fcc12e-b3e4-4064-a53e-3c250308aa2e") : secret "samples-operator-tls" not found
Jul 30 03:38:53.876510 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:53.876433    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:38:53.876510 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:53.876454    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65954c5f6d-hfc6f: secret "image-registry-tls" not found
Jul 30 03:38:53.876510 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:53.876483    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls podName:a5d33d98-cae0-4ef5-94f2-e133d0a2d51b nodeName:}" failed. No retries permitted until 2026-07-30 03:38:55.876472528 +0000 UTC m=+242.831107194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls") pod "image-registry-65954c5f6d-hfc6f" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b") : secret "image-registry-tls" not found
Jul 30 03:38:54.439499 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:54.439465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn" event={"ID":"a1e976d1-c889-442b-834d-fd69e9bca652","Type":"ContainerStarted","Data":"7673ae56cd16c3f79ad558ee2ffc2a3e149f733aad548492feb201e4ab53d93b"}
Jul 30 03:38:54.453334 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:54.453281    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-vx4rn" podStartSLOduration=1.197040627 podStartE2EDuration="2.453260213s" totalImagesPullingTime="1.256219586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:38:54.452992202 +0000 UTC m=+241.407626875" watchObservedRunningTime="2026-07-30 03:38:54.453260213 +0000 UTC m=+241.407894886"
Jul 30 03:38:55.444047 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:55.444008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx" event={"ID":"62d54750-5010-4150-8945-09a630b12105","Type":"ContainerStarted","Data":"99571e3f0962a5af458d0809d9303d8931f19f5c27e007077092a8f4884e2bb6"}
Jul 30 03:38:55.457245 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:55.457175    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx" podStartSLOduration=1.015400136 podStartE2EDuration="3.457155742s" totalImagesPullingTime="2.441755606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:38:55.457050656 +0000 UTC m=+242.411685329" watchObservedRunningTime="2026-07-30 03:38:55.457155742 +0000 UTC m=+242.411790414"
Jul 30 03:38:55.894594 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:55.894555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:55.894753 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:55.894607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:55.894753 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:55.894686    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:38:55.894835 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:55.894769    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls podName:b9fcc12e-b3e4-4064-a53e-3c250308aa2e nodeName:}" failed. No retries permitted until 2026-07-30 03:38:59.894747361 +0000 UTC m=+246.849382035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls") pod "cluster-samples-operator-6748489656-9qds7" (UID: "b9fcc12e-b3e4-4064-a53e-3c250308aa2e") : secret "samples-operator-tls" not found
Jul 30 03:38:55.894835 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:55.894695    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:38:55.894835 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:55.894789    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65954c5f6d-hfc6f: secret "image-registry-tls" not found
Jul 30 03:38:55.894940 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:55.894865    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls podName:a5d33d98-cae0-4ef5-94f2-e133d0a2d51b nodeName:}" failed. No retries permitted until 2026-07-30 03:38:59.894826384 +0000 UTC m=+246.849461035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls") pod "image-registry-65954c5f6d-hfc6f" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b") : secret "image-registry-tls" not found
Jul 30 03:38:57.671918 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.671882    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xdwvl"]
Jul 30 03:38:57.675216 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.675195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.678215 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.678195    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 03:38:57.678353 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.678201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-97fcc\""
Jul 30 03:38:57.678431 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.678369    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 03:38:57.678641 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.678622    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 03:38:57.678709 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.678623    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 03:38:57.710229 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.710200    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xdwvl"]
Jul 30 03:38:57.710388 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.710312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a796f82-3873-4cc3-92ee-b6bc2b80240a-crio-socket\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.710456 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.710387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a796f82-3873-4cc3-92ee-b6bc2b80240a-data-volume\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.710456 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.710416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.710564 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.710453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9j4c\" (UniqueName: \"kubernetes.io/projected/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-api-access-k9j4c\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.710564 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.710553    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.811702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.811662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a796f82-3873-4cc3-92ee-b6bc2b80240a-crio-socket\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.811900 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.811735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a796f82-3873-4cc3-92ee-b6bc2b80240a-data-volume\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.811900 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.811762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.811900 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.811792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9j4c\" (UniqueName: \"kubernetes.io/projected/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-api-access-k9j4c\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.811900 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.811803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a796f82-3873-4cc3-92ee-b6bc2b80240a-crio-socket\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.811900 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.811856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.811900 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:57.811880    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:38:57.812201 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:57.811958    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:38:58.311935601 +0000 UTC m=+245.266570254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : configmap "kube-rbac-proxy" not found
Jul 30 03:38:57.812201 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:57.811990    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:38:57.812201 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.812042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a796f82-3873-4cc3-92ee-b6bc2b80240a-data-volume\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:57.812201 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:57.812059    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:38:58.312041868 +0000 UTC m=+245.266676550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:38:57.819467 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:57.819442    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9j4c\" (UniqueName: \"kubernetes.io/projected/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-api-access-k9j4c\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:58.316074 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.316032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:58.316306 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.316093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:58.316306 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:58.316159    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:38:58.316306 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:58.316185    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:38:58.316306 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:58.316258    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:38:59.316220293 +0000 UTC m=+246.270854943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : configmap "kube-rbac-proxy" not found
Jul 30 03:38:58.316306 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:58.316276    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:38:59.31626881 +0000 UTC m=+246.270903460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:38:58.803647 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.803608    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-sbtm2"]
Jul 30 03:38:58.806972 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.806948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:58.808961 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.808938    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 30 03:38:58.809306 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.809290    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 30 03:38:58.809541 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.809525    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 30 03:38:58.809541 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.809533    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tbmsd\""
Jul 30 03:38:58.809667 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.809529    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 30 03:38:58.815989 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.815968    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-sbtm2"]
Jul 30 03:38:58.921719 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.921679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/292459a5-83ec-4993-9e02-e4022849e418-signing-cabundle\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:58.921719 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.921720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjph\" (UniqueName: \"kubernetes.io/projected/292459a5-83ec-4993-9e02-e4022849e418-kube-api-access-sbjph\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:58.921956 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:58.921759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/292459a5-83ec-4993-9e02-e4022849e418-signing-key\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:59.022954 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.022920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjph\" (UniqueName: \"kubernetes.io/projected/292459a5-83ec-4993-9e02-e4022849e418-kube-api-access-sbjph\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:59.023161 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.022988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/292459a5-83ec-4993-9e02-e4022849e418-signing-key\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:59.023161 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.023071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/292459a5-83ec-4993-9e02-e4022849e418-signing-cabundle\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:59.023767 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.023740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/292459a5-83ec-4993-9e02-e4022849e418-signing-cabundle\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:59.025356 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.025338    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/292459a5-83ec-4993-9e02-e4022849e418-signing-key\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:59.031173 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.031150    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjph\" (UniqueName: \"kubernetes.io/projected/292459a5-83ec-4993-9e02-e4022849e418-kube-api-access-sbjph\") pod \"service-ca-658d7b7c78-sbtm2\" (UID: \"292459a5-83ec-4993-9e02-e4022849e418\") " pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:59.116200 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.116110    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2"
Jul 30 03:38:59.237798 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.237769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-sbtm2"]
Jul 30 03:38:59.241018 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:38:59.240981    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292459a5_83ec_4993_9e02_e4022849e418.slice/crio-e969c1afebd3544a1ebd4e5cfc380ee876d9f2f95142194afd8e95178522ca9a WatchSource:0}: Error finding container e969c1afebd3544a1ebd4e5cfc380ee876d9f2f95142194afd8e95178522ca9a: Status 404 returned error can't find the container with id e969c1afebd3544a1ebd4e5cfc380ee876d9f2f95142194afd8e95178522ca9a
Jul 30 03:38:59.326589 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.326561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:59.326727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.326647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:38:59.326727 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.326678    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:38:59.326835 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.326734    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:39:01.326716488 +0000 UTC m=+248.281351138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : configmap "kube-rbac-proxy" not found
Jul 30 03:38:59.326835 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.326814    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:38:59.326941 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.326879    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:39:01.326862452 +0000 UTC m=+248.281497114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:38:59.456752 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.456718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2" event={"ID":"292459a5-83ec-4993-9e02-e4022849e418","Type":"ContainerStarted","Data":"6554fcfb4dc658469184a1bf250e4e5d853656c4db7667547095ec473689dfe6"}
Jul 30 03:38:59.456924 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.456767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2" event={"ID":"292459a5-83ec-4993-9e02-e4022849e418","Type":"ContainerStarted","Data":"e969c1afebd3544a1ebd4e5cfc380ee876d9f2f95142194afd8e95178522ca9a"}
Jul 30 03:38:59.457757 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.457742    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fltkz_48c8e946-a078-4b39-912c-fde7009abb28/dns-node-resolver/0.log"
Jul 30 03:38:59.473324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.473278    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-sbtm2" podStartSLOduration=1.473262503 podStartE2EDuration="1.473262503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:38:59.472461309 +0000 UTC m=+246.427095977" watchObservedRunningTime="2026-07-30 03:38:59.473262503 +0000 UTC m=+246.427897174"
Jul 30 03:38:59.931024 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.930998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:38:59.931438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:38:59.931041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:38:59.931438 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.931131    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:38:59.931438 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.931156    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:38:59.931438 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.931181    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65954c5f6d-hfc6f: secret "image-registry-tls" not found
Jul 30 03:38:59.931438 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.931185    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls podName:b9fcc12e-b3e4-4064-a53e-3c250308aa2e nodeName:}" failed. No retries permitted until 2026-07-30 03:39:07.931172039 +0000 UTC m=+254.885806688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls") pod "cluster-samples-operator-6748489656-9qds7" (UID: "b9fcc12e-b3e4-4064-a53e-3c250308aa2e") : secret "samples-operator-tls" not found
Jul 30 03:38:59.931438 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:38:59.931252    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls podName:a5d33d98-cae0-4ef5-94f2-e133d0a2d51b nodeName:}" failed. No retries permitted until 2026-07-30 03:39:07.93121513 +0000 UTC m=+254.885849784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls") pod "image-registry-65954c5f6d-hfc6f" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b") : secret "image-registry-tls" not found
Jul 30 03:39:00.657086 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:00.657055    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p2q9q_519e4a13-90e3-4d40-a097-cecb5171c54c/node-ca/0.log"
Jul 30 03:39:01.256917 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:01.256889    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ctdpt_58a0e78a-b220-4eb2-8435-7298c7e001f4/serve-healthcheck-canary/0.log"
Jul 30 03:39:01.344559 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:01.344527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:01.344763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:01.344597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:01.344763 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:01.344684    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:39:01.344763 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:01.344697    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:39:01.344763 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:01.344761    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:39:05.344744018 +0000 UTC m=+252.299378675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : configmap "kube-rbac-proxy" not found
Jul 30 03:39:01.344965 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:01.344777    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:39:05.344771328 +0000 UTC m=+252.299405977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:39:04.274672 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:04.274616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:39:04.275065 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:04.274773    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 30 03:39:04.275065 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:04.274843    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs podName:e5bd1a5c-9bd3-4874-991c-339874cf2ad0 nodeName:}" failed. No retries permitted until 2026-07-30 03:41:06.274826937 +0000 UTC m=+373.229461587 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs") pod "network-metrics-daemon-pc47f" (UID: "e5bd1a5c-9bd3-4874-991c-339874cf2ad0") : secret "metrics-daemon-secret" not found
Jul 30 03:39:05.384401 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:05.384357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:05.384830 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:05.384457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:05.384830 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:05.384486    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:39:05.384830 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:05.384565    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:39:13.384544115 +0000 UTC m=+260.339178769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : configmap "kube-rbac-proxy" not found
Jul 30 03:39:05.384830 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:05.384628    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:39:05.384830 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:05.384696    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls podName:2a796f82-3873-4cc3-92ee-b6bc2b80240a nodeName:}" failed. No retries permitted until 2026-07-30 03:39:13.384679082 +0000 UTC m=+260.339313734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xdwvl" (UID: "2a796f82-3873-4cc3-92ee-b6bc2b80240a") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:39:08.005351 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.005311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:39:08.005729 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.005380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:39:08.007910 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.007879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"image-registry-65954c5f6d-hfc6f\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") " pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:39:08.008013 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.007919    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9fcc12e-b3e4-4064-a53e-3c250308aa2e-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9qds7\" (UID: \"b9fcc12e-b3e4-4064-a53e-3c250308aa2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:39:08.078411 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.078375    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j56jf\""
Jul 30 03:39:08.084216 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.084197    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vqpns\""
Jul 30 03:39:08.087435 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.087421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"
Jul 30 03:39:08.093290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.093268    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:39:08.231557 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.231520    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65954c5f6d-hfc6f"]
Jul 30 03:39:08.235704 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:08.235671    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d33d98_cae0_4ef5_94f2_e133d0a2d51b.slice/crio-105a9d6183c568896ee13fa54a812fa1ed52ea3c264ace3f64015b7347360665 WatchSource:0}: Error finding container 105a9d6183c568896ee13fa54a812fa1ed52ea3c264ace3f64015b7347360665: Status 404 returned error can't find the container with id 105a9d6183c568896ee13fa54a812fa1ed52ea3c264ace3f64015b7347360665
Jul 30 03:39:08.248268 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.248168    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7"]
Jul 30 03:39:08.488347 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.488304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f" event={"ID":"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b","Type":"ContainerStarted","Data":"816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f"}
Jul 30 03:39:08.488548 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.488366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f" event={"ID":"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b","Type":"ContainerStarted","Data":"105a9d6183c568896ee13fa54a812fa1ed52ea3c264ace3f64015b7347360665"}
Jul 30 03:39:08.488548 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.488391    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:39:08.489496 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.489460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7" event={"ID":"b9fcc12e-b3e4-4064-a53e-3c250308aa2e","Type":"ContainerStarted","Data":"7b50919ea134f3f360bc06fdd25e561acb49cef400178c849d2750dbc20f4e13"}
Jul 30 03:39:08.508463 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:08.508361    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f" podStartSLOduration=16.508343251 podStartE2EDuration="16.508343251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:08.507072953 +0000 UTC m=+255.461707625" watchObservedRunningTime="2026-07-30 03:39:08.508343251 +0000 UTC m=+255.462977923"
Jul 30 03:39:10.497765 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:10.497720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7" event={"ID":"b9fcc12e-b3e4-4064-a53e-3c250308aa2e","Type":"ContainerStarted","Data":"ed449781a3f5d427d3717fd8e4a9f54ef7df83f4458dc193a8c09689bc1b0601"}
Jul 30 03:39:10.498135 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:10.497795    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7" event={"ID":"b9fcc12e-b3e4-4064-a53e-3c250308aa2e","Type":"ContainerStarted","Data":"7f9e7d82623dd8a58db894c31e4907572cfcab394e3d5a85539d47de629821fd"}
Jul 30 03:39:10.513819 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:10.513754    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9qds7" podStartSLOduration=16.532011374 podStartE2EDuration="18.513731837s" totalImagesPullingTime="1.981720463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:10.512969337 +0000 UTC m=+257.467604007" watchObservedRunningTime="2026-07-30 03:39:10.513731837 +0000 UTC m=+257.468366701"
Jul 30 03:39:13.455398 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:13.455355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:13.455771 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:13.455447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:13.455964 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:13.455947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a796f82-3873-4cc3-92ee-b6bc2b80240a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:13.457818 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:13.457787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a796f82-3873-4cc3-92ee-b6bc2b80240a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xdwvl\" (UID: \"2a796f82-3873-4cc3-92ee-b6bc2b80240a\") " pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:13.584249 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:13.584214    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xdwvl"
Jul 30 03:39:13.713064 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:13.712971    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xdwvl"]
Jul 30 03:39:13.716555 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:13.716522    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a796f82_3873_4cc3_92ee_b6bc2b80240a.slice/crio-b770a1afd9e6cbf2d79e4b039f1114f58e8f940182d771e330eb62bc588f66c8 WatchSource:0}: Error finding container b770a1afd9e6cbf2d79e4b039f1114f58e8f940182d771e330eb62bc588f66c8: Status 404 returned error can't find the container with id b770a1afd9e6cbf2d79e4b039f1114f58e8f940182d771e330eb62bc588f66c8
Jul 30 03:39:14.513441 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:14.513395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xdwvl" event={"ID":"2a796f82-3873-4cc3-92ee-b6bc2b80240a","Type":"ContainerStarted","Data":"b770a1afd9e6cbf2d79e4b039f1114f58e8f940182d771e330eb62bc588f66c8"}
Jul 30 03:39:16.924206 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:16.924108    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65954c5f6d-hfc6f"]
Jul 30 03:39:16.968215 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:16.968180    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5879b768bb-n7g4g"]
Jul 30 03:39:16.990023 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:16.989980    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5879b768bb-n7g4g"]
Jul 30 03:39:16.990181 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:16.990117    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.089211 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.089179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4565b6b-c6ff-474b-8208-b83400bce0c4-registry-certificates\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.089211 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.089217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4565b6b-c6ff-474b-8208-b83400bce0c4-trusted-ca\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.089465 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.089337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4565b6b-c6ff-474b-8208-b83400bce0c4-image-registry-private-configuration\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.089465 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.089454    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4565b6b-c6ff-474b-8208-b83400bce0c4-ca-trust-extracted\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.089543 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.089499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-bound-sa-token\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.089585 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.089563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjbj\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-kube-api-access-4bjbj\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.089628 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.089592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-registry-tls\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.089675 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.089625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4565b6b-c6ff-474b-8208-b83400bce0c4-installation-pull-secrets\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.190767 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.190677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjbj\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-kube-api-access-4bjbj\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.190767 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.190729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-registry-tls\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.190767 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.190759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4565b6b-c6ff-474b-8208-b83400bce0c4-installation-pull-secrets\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.191040 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.190790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4565b6b-c6ff-474b-8208-b83400bce0c4-registry-certificates\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.191040 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.190808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4565b6b-c6ff-474b-8208-b83400bce0c4-trusted-ca\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.191040 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.190853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4565b6b-c6ff-474b-8208-b83400bce0c4-image-registry-private-configuration\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.191040 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.190918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4565b6b-c6ff-474b-8208-b83400bce0c4-ca-trust-extracted\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.191040 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.190942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-bound-sa-token\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.192791 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.192703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4565b6b-c6ff-474b-8208-b83400bce0c4-ca-trust-extracted\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.193573 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.193332    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4565b6b-c6ff-474b-8208-b83400bce0c4-trusted-ca\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.193700 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.193511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4565b6b-c6ff-474b-8208-b83400bce0c4-registry-certificates\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.198495 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.198419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4565b6b-c6ff-474b-8208-b83400bce0c4-image-registry-private-configuration\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.198974 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.198874    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-registry-tls\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.199617 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.199594    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4565b6b-c6ff-474b-8208-b83400bce0c4-installation-pull-secrets\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.204997 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.204971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-bound-sa-token\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.205932 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.205911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjbj\" (UniqueName: \"kubernetes.io/projected/d4565b6b-c6ff-474b-8208-b83400bce0c4-kube-api-access-4bjbj\") pod \"image-registry-5879b768bb-n7g4g\" (UID: \"d4565b6b-c6ff-474b-8208-b83400bce0c4\") " pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:17.302721 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:17.302690    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:18.595057 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:18.595025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5879b768bb-n7g4g"]
Jul 30 03:39:18.598984 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:18.598958    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4565b6b_c6ff_474b_8208_b83400bce0c4.slice/crio-a5900bf00cce65ecbb08caa768f69f4e4fcd10369182a92155e284b8b0a544ff WatchSource:0}: Error finding container a5900bf00cce65ecbb08caa768f69f4e4fcd10369182a92155e284b8b0a544ff: Status 404 returned error can't find the container with id a5900bf00cce65ecbb08caa768f69f4e4fcd10369182a92155e284b8b0a544ff
Jul 30 03:39:19.533037 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:19.532980    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g" event={"ID":"d4565b6b-c6ff-474b-8208-b83400bce0c4","Type":"ContainerStarted","Data":"4f985e6ce4135370b9e1349d2043c7a129e86395a003a3e9f8cbef5d7c8ddb63"}
Jul 30 03:39:19.533242 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:19.533123    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:19.533242 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:19.533146    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g" event={"ID":"d4565b6b-c6ff-474b-8208-b83400bce0c4","Type":"ContainerStarted","Data":"a5900bf00cce65ecbb08caa768f69f4e4fcd10369182a92155e284b8b0a544ff"}
Jul 30 03:39:19.534983 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:19.534959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xdwvl" event={"ID":"2a796f82-3873-4cc3-92ee-b6bc2b80240a","Type":"ContainerStarted","Data":"8e67fcb2b807b4595b51e2216ec6647643060d2434d08b449805b573a31eefb4"}
Jul 30 03:39:19.552822 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:19.552766    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g" podStartSLOduration=3.552749418 podStartE2EDuration="3.552749418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:19.552244809 +0000 UTC m=+266.506879486" watchObservedRunningTime="2026-07-30 03:39:19.552749418 +0000 UTC m=+266.507384066"
Jul 30 03:39:22.547022 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:22.546934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xdwvl" event={"ID":"2a796f82-3873-4cc3-92ee-b6bc2b80240a","Type":"ContainerStarted","Data":"4e6948dc8a508e6039c18489d822dcecd9c51c10db32506425332052d3920c4f"}
Jul 30 03:39:26.561817 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:26.561782    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xdwvl" event={"ID":"2a796f82-3873-4cc3-92ee-b6bc2b80240a","Type":"ContainerStarted","Data":"b812582a765c391f03b8ea2ee30c49e48006191adf9670835b7206ebd6b750be"}
Jul 30 03:39:26.577657 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:26.577548    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xdwvl" podStartSLOduration=17.841229206 podStartE2EDuration="29.577530052s" totalImagesPullingTime="11.736300846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:38:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:26.577354504 +0000 UTC m=+273.531989189" watchObservedRunningTime="2026-07-30 03:39:26.577530052 +0000 UTC m=+273.532164723"
Jul 30 03:39:26.930208 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:26.930170    2560 patch_prober.go:28] interesting pod/image-registry-65954c5f6d-hfc6f container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 03:39:26.930416 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:26.930250    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f" podUID="a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:39:27.943820 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:27.943784    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"]
Jul 30 03:39:27.949736 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:27.949716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"
Jul 30 03:39:27.951502 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:27.951467    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 30 03:39:27.951629 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:27.951537    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bz994\""
Jul 30 03:39:27.956002 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:27.955977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"]
Jul 30 03:39:28.085084 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:28.085047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9236e3ee-cc6a-4a67-ad5d-0fa156d706b6-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-rpzcc\" (UID: \"9236e3ee-cc6a-4a67-ad5d-0fa156d706b6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"
Jul 30 03:39:28.185499 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:28.185458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9236e3ee-cc6a-4a67-ad5d-0fa156d706b6-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-rpzcc\" (UID: \"9236e3ee-cc6a-4a67-ad5d-0fa156d706b6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"
Jul 30 03:39:28.187862 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:28.187841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9236e3ee-cc6a-4a67-ad5d-0fa156d706b6-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-rpzcc\" (UID: \"9236e3ee-cc6a-4a67-ad5d-0fa156d706b6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"
Jul 30 03:39:28.260390 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:28.260298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"
Jul 30 03:39:28.414698 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:28.414664    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"]
Jul 30 03:39:28.418029 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:28.417998    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9236e3ee_cc6a_4a67_ad5d_0fa156d706b6.slice/crio-e837a208c93f6cf5bf235bbf34e27d1b2f26a0b0885e30a49c99df680cdef4af WatchSource:0}: Error finding container e837a208c93f6cf5bf235bbf34e27d1b2f26a0b0885e30a49c99df680cdef4af: Status 404 returned error can't find the container with id e837a208c93f6cf5bf235bbf34e27d1b2f26a0b0885e30a49c99df680cdef4af
Jul 30 03:39:28.573385 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:28.573294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc" event={"ID":"9236e3ee-cc6a-4a67-ad5d-0fa156d706b6","Type":"ContainerStarted","Data":"e837a208c93f6cf5bf235bbf34e27d1b2f26a0b0885e30a49c99df680cdef4af"}
Jul 30 03:39:30.581245 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:30.581197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc" event={"ID":"9236e3ee-cc6a-4a67-ad5d-0fa156d706b6","Type":"ContainerStarted","Data":"02b8944fc421cc02ccb190c7ef7b48e2ea7ac7f28c954cba2e3a0de55375862c"}
Jul 30 03:39:30.581668 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:30.581360    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"
Jul 30 03:39:30.586128 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:30.586104    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc"
Jul 30 03:39:30.595696 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:30.595650    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rpzcc" podStartSLOduration=2.350861625 podStartE2EDuration="3.595636224s" totalImagesPullingTime="1.244774599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:30.594900492 +0000 UTC m=+277.549535163" watchObservedRunningTime="2026-07-30 03:39:30.595636224 +0000 UTC m=+277.550270910"
Jul 30 03:39:35.229495 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:35.229443    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7t5ct" podUID="d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa"
Jul 30 03:39:35.596781 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:35.596706    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:39:36.376692 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.376650    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"]
Jul 30 03:39:36.380566 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.380546    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.383117 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.383090    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 30 03:39:36.383263 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.383159    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-q8rsf\""
Jul 30 03:39:36.383327 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.383098    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 03:39:36.383327 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.383095    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 03:39:36.383500 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.383477    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 30 03:39:36.383615 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.383524    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 03:39:36.391825 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.391801    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"]
Jul 30 03:39:36.395305 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.395277    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"]
Jul 30 03:39:36.398932 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.398913    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.401538 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.401511    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x5kkp\""
Jul 30 03:39:36.401663 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.401562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 30 03:39:36.401733 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.401510    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 30 03:39:36.401805 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.401786    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 30 03:39:36.422751 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.422720    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"]
Jul 30 03:39:36.426004 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.425969    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s42w5"]
Jul 30 03:39:36.429688 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.429667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.432127 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.432106    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 03:39:36.432265 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.432107    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 03:39:36.432422 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.432404    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 03:39:36.432722 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.432707    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6j2hf\""
Jul 30 03:39:36.448551 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjg8\" (UniqueName: \"kubernetes.io/projected/5be24e71-72df-46b3-8aff-844f4355f31a-kube-api-access-ghjg8\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.448701 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.448701 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.448701 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27rk\" (UniqueName: \"kubernetes.io/projected/991402de-3dde-4b38-8878-d54c9eae6f9f-kube-api-access-d27rk\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.448701 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.448701 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448701    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc36ada-4a95-4b5a-a34f-ca97991de624-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-root\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbc36ada-4a95-4b5a-a34f-ca97991de624-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be24e71-72df-46b3-8aff-844f4355f31a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-sys\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-textfile\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.448948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/991402de-3dde-4b38-8878-d54c9eae6f9f-metrics-client-ca\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.449444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.448990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-wtmp\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.449444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.449029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zlkg\" (UniqueName: \"kubernetes.io/projected/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-api-access-6zlkg\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.449444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.449055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-tls\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.449444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.449115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.550364 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-wtmp\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.550364 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zlkg\" (UniqueName: \"kubernetes.io/projected/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-api-access-6zlkg\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.550625 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-tls\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.550625 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.550625 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjg8\" (UniqueName: \"kubernetes.io/projected/5be24e71-72df-46b3-8aff-844f4355f31a-kube-api-access-ghjg8\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.550625 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.550625 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.550625 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550603    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27rk\" (UniqueName: \"kubernetes.io/projected/991402de-3dde-4b38-8878-d54c9eae6f9f-kube-api-access-d27rk\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.550625 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-wtmp\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.550981 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.550981 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.550665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.550981 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:36.550952    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 30 03:39:36.551121 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:36.551030    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-tls podName:5be24e71-72df-46b3-8aff-844f4355f31a nodeName:}" failed. No retries permitted until 2026-07-30 03:39:37.051010137 +0000 UTC m=+284.005644802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-sv77j" (UID: "5be24e71-72df-46b3-8aff-844f4355f31a") : secret "openshift-state-metrics-tls" not found
Jul 30 03:39:36.551121 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc36ada-4a95-4b5a-a34f-ca97991de624-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.551261 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-root\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.551261 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbc36ada-4a95-4b5a-a34f-ca97991de624-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.551261 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be24e71-72df-46b3-8aff-844f4355f31a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.551414 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-sys\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.551414 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-root\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.551414 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-textfile\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.551414 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.551414 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.551414 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.551713 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/991402de-3dde-4b38-8878-d54c9eae6f9f-metrics-client-ca\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.551713 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-textfile\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.551713 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbc36ada-4a95-4b5a-a34f-ca97991de624-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.551713 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991402de-3dde-4b38-8878-d54c9eae6f9f-sys\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.551903 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.551875    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc36ada-4a95-4b5a-a34f-ca97991de624-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.552114 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.552087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/991402de-3dde-4b38-8878-d54c9eae6f9f-metrics-client-ca\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.552225 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.552201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.552426 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.552263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be24e71-72df-46b3-8aff-844f4355f31a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.553528 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.553494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-tls\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.554317 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.554297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/991402de-3dde-4b38-8878-d54c9eae6f9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.554398 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.554326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.555043 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.554969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.555286 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.555265    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.559356 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.559331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zlkg\" (UniqueName: \"kubernetes.io/projected/fbc36ada-4a95-4b5a-a34f-ca97991de624-kube-api-access-6zlkg\") pod \"kube-state-metrics-56d9f944b4-llplc\" (UID: \"fbc36ada-4a95-4b5a-a34f-ca97991de624\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.559479 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.559332    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjg8\" (UniqueName: \"kubernetes.io/projected/5be24e71-72df-46b3-8aff-844f4355f31a-kube-api-access-ghjg8\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:36.559597 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.559579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27rk\" (UniqueName: \"kubernetes.io/projected/991402de-3dde-4b38-8878-d54c9eae6f9f-kube-api-access-d27rk\") pod \"node-exporter-s42w5\" (UID: \"991402de-3dde-4b38-8878-d54c9eae6f9f\") " pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.710035 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.709994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"
Jul 30 03:39:36.741659 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.741628    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s42w5"
Jul 30 03:39:36.752887 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:36.752849    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991402de_3dde_4b38_8878_d54c9eae6f9f.slice/crio-9399a8aa00f38ba7b0f0d464ac6368d78070cbb8e5e8cf2acbf90421bf3448ae WatchSource:0}: Error finding container 9399a8aa00f38ba7b0f0d464ac6368d78070cbb8e5e8cf2acbf90421bf3448ae: Status 404 returned error can't find the container with id 9399a8aa00f38ba7b0f0d464ac6368d78070cbb8e5e8cf2acbf90421bf3448ae
Jul 30 03:39:36.852733 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.852701    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-llplc"]
Jul 30 03:39:36.855713 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:36.855686    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc36ada_4a95_4b5a_a34f_ca97991de624.slice/crio-34845cb7317fdf394e70398dc171876bbee52d99fc8fcb945777f629f93600e5 WatchSource:0}: Error finding container 34845cb7317fdf394e70398dc171876bbee52d99fc8fcb945777f629f93600e5: Status 404 returned error can't find the container with id 34845cb7317fdf394e70398dc171876bbee52d99fc8fcb945777f629f93600e5
Jul 30 03:39:36.929000 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.928959    2560 patch_prober.go:28] interesting pod/image-registry-65954c5f6d-hfc6f container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 03:39:36.929167 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:36.929009    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f" podUID="a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:39:37.056036 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.055936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:37.058710 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.058680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5be24e71-72df-46b3-8aff-844f4355f31a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-sv77j\" (UID: \"5be24e71-72df-46b3-8aff-844f4355f31a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:37.291314 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.291278    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"
Jul 30 03:39:37.308489 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.308407    2560 patch_prober.go:28] interesting pod/image-registry-5879b768bb-n7g4g container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 03:39:37.308489 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.308467    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g" podUID="d4565b6b-c6ff-474b-8208-b83400bce0c4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:39:37.449836 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.449803    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j"]
Jul 30 03:39:37.452833 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:37.452808    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be24e71_72df_46b3_8aff_844f4355f31a.slice/crio-714c5a6553419ce60025b0e6356606aa67f7e04add71ecbedb718ba14530966d WatchSource:0}: Error finding container 714c5a6553419ce60025b0e6356606aa67f7e04add71ecbedb718ba14530966d: Status 404 returned error can't find the container with id 714c5a6553419ce60025b0e6356606aa67f7e04add71ecbedb718ba14530966d
Jul 30 03:39:37.615821 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.615644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s42w5" event={"ID":"991402de-3dde-4b38-8878-d54c9eae6f9f","Type":"ContainerStarted","Data":"9399a8aa00f38ba7b0f0d464ac6368d78070cbb8e5e8cf2acbf90421bf3448ae"}
Jul 30 03:39:37.617303 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.617270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc" event={"ID":"fbc36ada-4a95-4b5a-a34f-ca97991de624","Type":"ContainerStarted","Data":"34845cb7317fdf394e70398dc171876bbee52d99fc8fcb945777f629f93600e5"}
Jul 30 03:39:37.618744 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:37.618693    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j" event={"ID":"5be24e71-72df-46b3-8aff-844f4355f31a","Type":"ContainerStarted","Data":"714c5a6553419ce60025b0e6356606aa67f7e04add71ecbedb718ba14530966d"}
Jul 30 03:39:38.367712 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:38.367618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:39:38.370616 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:38.370590    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa-metrics-tls\") pod \"dns-default-7t5ct\" (UID: \"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa\") " pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:39:38.602202 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:38.602173    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nmkwv\""
Jul 30 03:39:38.607745 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:38.607718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:39:38.625414 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:38.625336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j" event={"ID":"5be24e71-72df-46b3-8aff-844f4355f31a","Type":"ContainerStarted","Data":"9aa078a3186d183136f96be236791a066e12bdc8bec4b34f4d531894788bccb8"}
Jul 30 03:39:38.625548 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:38.625417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j" event={"ID":"5be24e71-72df-46b3-8aff-844f4355f31a","Type":"ContainerStarted","Data":"b9045db4db442d4583e48996d33a8a317c37169de84115d7a560a3893a993ad3"}
Jul 30 03:39:38.627031 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:38.627001    2560 generic.go:356] "Generic (PLEG): container finished" podID="991402de-3dde-4b38-8878-d54c9eae6f9f" containerID="2f7f2d277f4b1e57d6ab32560dda2376e43e96ecaeb7eb1e2c683fc246f589fe" exitCode=0
Jul 30 03:39:38.627151 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:38.627076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s42w5" event={"ID":"991402de-3dde-4b38-8878-d54c9eae6f9f","Type":"ContainerDied","Data":"2f7f2d277f4b1e57d6ab32560dda2376e43e96ecaeb7eb1e2c683fc246f589fe"}
Jul 30 03:39:39.150360 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.150107    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7t5ct"]
Jul 30 03:39:39.159394 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:39.159300    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c014a2_8b6d_4325_b3f4_f3d0c6534cfa.slice/crio-e81628fc8801a578548be5cba325fbf8404bacf5979abb80b870123b5bcab397 WatchSource:0}: Error finding container e81628fc8801a578548be5cba325fbf8404bacf5979abb80b870123b5bcab397: Status 404 returned error can't find the container with id e81628fc8801a578548be5cba325fbf8404bacf5979abb80b870123b5bcab397
Jul 30 03:39:39.633695 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.633657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s42w5" event={"ID":"991402de-3dde-4b38-8878-d54c9eae6f9f","Type":"ContainerStarted","Data":"be9f0907d561a9a87b4de00386bc08a36197791f69f733d46e78f7407f34e0ab"}
Jul 30 03:39:39.634144 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.633725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s42w5" event={"ID":"991402de-3dde-4b38-8878-d54c9eae6f9f","Type":"ContainerStarted","Data":"d7c707235e8ad8708e34e41a22872266bc33947b00a661872ddf5ab74bcfaeb0"}
Jul 30 03:39:39.636303 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.636270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc" event={"ID":"fbc36ada-4a95-4b5a-a34f-ca97991de624","Type":"ContainerStarted","Data":"d3f91f5fbd4f372d868a842d7c1474ea02aed8f834316431854f29ff0ee793a3"}
Jul 30 03:39:39.636438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.636372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc" event={"ID":"fbc36ada-4a95-4b5a-a34f-ca97991de624","Type":"ContainerStarted","Data":"b3a2cd2073a01480a65d4d025411e97438cb2ddac727d1a7e9e1ad7d1c2684fb"}
Jul 30 03:39:39.636438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.636403    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc" event={"ID":"fbc36ada-4a95-4b5a-a34f-ca97991de624","Type":"ContainerStarted","Data":"676620bf876469a045d3ca7c99e4a7a1badad67b769a3569849a51523e64f7de"}
Jul 30 03:39:39.638709 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.638681    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j" event={"ID":"5be24e71-72df-46b3-8aff-844f4355f31a","Type":"ContainerStarted","Data":"e652cb1a5b183f2a921aec9d42c3a51935b7fe2351df1dfb4934d48f480ea022"}
Jul 30 03:39:39.639956 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.639929    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7t5ct" event={"ID":"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa","Type":"ContainerStarted","Data":"e81628fc8801a578548be5cba325fbf8404bacf5979abb80b870123b5bcab397"}
Jul 30 03:39:39.652543 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.652485    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s42w5" podStartSLOduration=2.618336929 podStartE2EDuration="3.652469874s" totalImagesPullingTime="966.650509ms" totalInitContainerRuntime="67.482436ms" isStatefulPod=true podCreationTimestamp="2026-07-30 03:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:39.651140293 +0000 UTC m=+286.605774964" watchObservedRunningTime="2026-07-30 03:39:39.652469874 +0000 UTC m=+286.607104544"
Jul 30 03:39:39.669141 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.669042    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-sv77j" podStartSLOduration=2.496242493 podStartE2EDuration="3.669013734s" totalImagesPullingTime="1.172771241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:39.667914195 +0000 UTC m=+286.622548868" watchObservedRunningTime="2026-07-30 03:39:39.669013734 +0000 UTC m=+286.623648406"
Jul 30 03:39:39.685967 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:39.685897    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-llplc" podStartSLOduration=1.5542618209999999 podStartE2EDuration="3.68587682s" totalImagesPullingTime="2.131614999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:39.685039135 +0000 UTC m=+286.639673788" watchObservedRunningTime="2026-07-30 03:39:39.68587682 +0000 UTC m=+286.640511491"
Jul 30 03:39:40.544486 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.544451    2560 patch_prober.go:28] interesting pod/image-registry-5879b768bb-n7g4g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 03:39:40.544680 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.544513    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g" podUID="d4565b6b-c6ff-474b-8208-b83400bce0c4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:39:40.721598 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.721562    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c57d6c8-bbgsg"]
Jul 30 03:39:40.726073 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.726051    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.728965 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.728939    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 30 03:39:40.729165 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.728999    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ps7qx\""
Jul 30 03:39:40.729291 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.729194    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 30 03:39:40.729291 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.729204    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 30 03:39:40.729291 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.728999    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5dtkuc1lp140n\""
Jul 30 03:39:40.729291 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.729055    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 30 03:39:40.739649 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.739626    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c57d6c8-bbgsg"]
Jul 30 03:39:40.790572 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.790543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45cc857f-c973-46f1-8cb3-7ae3efe45767-audit-log\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.790572 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.790572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45cc857f-c973-46f1-8cb3-7ae3efe45767-metrics-server-audit-profiles\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.790768 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.790592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-secret-metrics-server-client-certs\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.790768 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.790619    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45cc857f-c973-46f1-8cb3-7ae3efe45767-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.790768 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.790637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494kz\" (UniqueName: \"kubernetes.io/projected/45cc857f-c973-46f1-8cb3-7ae3efe45767-kube-api-access-494kz\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.790768 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.790659    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-secret-metrics-server-tls\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.790768 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.790722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-client-ca-bundle\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.892342 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.891656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-client-ca-bundle\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.892342 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.891729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45cc857f-c973-46f1-8cb3-7ae3efe45767-audit-log\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.892342 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.891763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45cc857f-c973-46f1-8cb3-7ae3efe45767-metrics-server-audit-profiles\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.892342 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.891791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-secret-metrics-server-client-certs\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.892342 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.891815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45cc857f-c973-46f1-8cb3-7ae3efe45767-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.892342 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.891838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494kz\" (UniqueName: \"kubernetes.io/projected/45cc857f-c973-46f1-8cb3-7ae3efe45767-kube-api-access-494kz\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.892342 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.891871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-secret-metrics-server-tls\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.892805 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.892676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45cc857f-c973-46f1-8cb3-7ae3efe45767-audit-log\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.893185 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.893149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45cc857f-c973-46f1-8cb3-7ae3efe45767-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.893439 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.893415    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45cc857f-c973-46f1-8cb3-7ae3efe45767-metrics-server-audit-profiles\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.895059 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.895034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-client-ca-bundle\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.895059 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.895051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-secret-metrics-server-tls\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.895368 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.895345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45cc857f-c973-46f1-8cb3-7ae3efe45767-secret-metrics-server-client-certs\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:40.901782 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:40.901759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-494kz\" (UniqueName: \"kubernetes.io/projected/45cc857f-c973-46f1-8cb3-7ae3efe45767-kube-api-access-494kz\") pod \"metrics-server-5c57d6c8-bbgsg\" (UID: \"45cc857f-c973-46f1-8cb3-7ae3efe45767\") " pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:41.039420 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.039380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:39:41.176661 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.176622    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"]
Jul 30 03:39:41.181180 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.181154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:41.182414 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.182385    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c57d6c8-bbgsg"]
Jul 30 03:39:41.183096 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.183078    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 30 03:39:41.183180 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.183103    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gx564\""
Jul 30 03:39:41.186215 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:41.186189    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cc857f_c973_46f1_8cb3_7ae3efe45767.slice/crio-f53508acca1d2e390c0535ac5b4d21c92149a4655ab4d29904c0f359362e6b83 WatchSource:0}: Error finding container f53508acca1d2e390c0535ac5b4d21c92149a4655ab4d29904c0f359362e6b83: Status 404 returned error can't find the container with id f53508acca1d2e390c0535ac5b4d21c92149a4655ab4d29904c0f359362e6b83
Jul 30 03:39:41.189072 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.188989    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"]
Jul 30 03:39:41.294609 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.294558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbt6\" (UniqueName: \"kubernetes.io/projected/f99ec3fe-a2c1-441a-8361-3f8f7fdcb886-kube-api-access-4jbt6\") pod \"monitoring-plugin-68b7fc9574-kdm4d\" (UID: \"f99ec3fe-a2c1-441a-8361-3f8f7fdcb886\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:41.294609 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.294616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f99ec3fe-a2c1-441a-8361-3f8f7fdcb886-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-kdm4d\" (UID: \"f99ec3fe-a2c1-441a-8361-3f8f7fdcb886\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:41.395822 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.395794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbt6\" (UniqueName: \"kubernetes.io/projected/f99ec3fe-a2c1-441a-8361-3f8f7fdcb886-kube-api-access-4jbt6\") pod \"monitoring-plugin-68b7fc9574-kdm4d\" (UID: \"f99ec3fe-a2c1-441a-8361-3f8f7fdcb886\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:41.395971 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.395839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f99ec3fe-a2c1-441a-8361-3f8f7fdcb886-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-kdm4d\" (UID: \"f99ec3fe-a2c1-441a-8361-3f8f7fdcb886\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:41.398424 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.398403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f99ec3fe-a2c1-441a-8361-3f8f7fdcb886-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-kdm4d\" (UID: \"f99ec3fe-a2c1-441a-8361-3f8f7fdcb886\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:41.403345 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.403324    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbt6\" (UniqueName: \"kubernetes.io/projected/f99ec3fe-a2c1-441a-8361-3f8f7fdcb886-kube-api-access-4jbt6\") pod \"monitoring-plugin-68b7fc9574-kdm4d\" (UID: \"f99ec3fe-a2c1-441a-8361-3f8f7fdcb886\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:41.492432 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.492396    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:41.623870 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.623841    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5557d7d88f-dljgh"]
Jul 30 03:39:41.629001 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.628926    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"]
Jul 30 03:39:41.629094 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.629019    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.631740 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:41.631704    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99ec3fe_a2c1_441a_8361_3f8f7fdcb886.slice/crio-956976330bdac17a7090e344951f1a215264ce4de97cd11cb4b72f48afda7b74 WatchSource:0}: Error finding container 956976330bdac17a7090e344951f1a215264ce4de97cd11cb4b72f48afda7b74: Status 404 returned error can't find the container with id 956976330bdac17a7090e344951f1a215264ce4de97cd11cb4b72f48afda7b74
Jul 30 03:39:41.631868 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.631850    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 30 03:39:41.632487 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.632468    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 30 03:39:41.632588 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.632561    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 30 03:39:41.632588 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.632574    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 30 03:39:41.632864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.632851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 30 03:39:41.632917 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.632884    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6vr7l\""
Jul 30 03:39:41.638343 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.638322    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 30 03:39:41.642769 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.642749    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5557d7d88f-dljgh"]
Jul 30 03:39:41.649809 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.649740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7t5ct" event={"ID":"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa","Type":"ContainerStarted","Data":"13e9d756ba2599521aaca99b7ecbda9679cc2dcab9efeb711bf3f06ca14c7459"}
Jul 30 03:39:41.649809 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.649793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7t5ct" event={"ID":"d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa","Type":"ContainerStarted","Data":"c877a97f33007ced6f4e517fa481a45aab52a85428cec37e7a6674e0f6bddccc"}
Jul 30 03:39:41.649976 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.649879    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:39:41.651098 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.651074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg" event={"ID":"45cc857f-c973-46f1-8cb3-7ae3efe45767","Type":"ContainerStarted","Data":"f53508acca1d2e390c0535ac5b4d21c92149a4655ab4d29904c0f359362e6b83"}
Jul 30 03:39:41.652155 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.652134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d" event={"ID":"f99ec3fe-a2c1-441a-8361-3f8f7fdcb886","Type":"ContainerStarted","Data":"956976330bdac17a7090e344951f1a215264ce4de97cd11cb4b72f48afda7b74"}
Jul 30 03:39:41.684315 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.684261    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7t5ct" podStartSLOduration=252.147457816 podStartE2EDuration="4m13.684230118s" totalImagesPullingTime="1.536772302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:41.682437341 +0000 UTC m=+288.637072011" watchObservedRunningTime="2026-07-30 03:39:41.684230118 +0000 UTC m=+288.638864789"
Jul 30 03:39:41.698981 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.698948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-serving-certs-ca-bundle\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.699144 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.699023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.699144 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.699091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlrb\" (UniqueName: \"kubernetes.io/projected/76f62d5f-1f8f-43b8-95d0-950a657878ef-kube-api-access-nqlrb\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.699290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.699140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-federate-client-tls\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.699290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.699182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-secret-telemeter-client\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.699290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.699205    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-metrics-client-ca\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.699290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.699284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.699489 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.699362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-telemeter-client-tls\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.800251 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.800202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.800727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.800285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-telemeter-client-tls\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.800727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.800342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-serving-certs-ca-bundle\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.800727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.800386    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.800727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.800421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlrb\" (UniqueName: \"kubernetes.io/projected/76f62d5f-1f8f-43b8-95d0-950a657878ef-kube-api-access-nqlrb\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.800727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.800456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-federate-client-tls\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.800727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.800486    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-secret-telemeter-client\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.800727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.800530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-metrics-client-ca\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.801408 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.801360    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-metrics-client-ca\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.801890 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.801866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-serving-certs-ca-bundle\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.802384 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.802344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f62d5f-1f8f-43b8-95d0-950a657878ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.803355 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.803330    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.804203 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.804140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-secret-telemeter-client\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.804203 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.804153    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-federate-client-tls\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.804371 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.804258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76f62d5f-1f8f-43b8-95d0-950a657878ef-telemeter-client-tls\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.809446 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.809425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlrb\" (UniqueName: \"kubernetes.io/projected/76f62d5f-1f8f-43b8-95d0-950a657878ef-kube-api-access-nqlrb\") pod \"telemeter-client-5557d7d88f-dljgh\" (UID: \"76f62d5f-1f8f-43b8-95d0-950a657878ef\") " pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.939741 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.939659    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh"
Jul 30 03:39:41.944974 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:41.944932    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f" podUID="a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" containerName="registry" containerID="cri-o://816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f" gracePeriod=30
Jul 30 03:39:42.099559 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.099485    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5557d7d88f-dljgh"]
Jul 30 03:39:42.104526 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:42.104494    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f62d5f_1f8f_43b8_95d0_950a657878ef.slice/crio-468f3cd7869773fa7f7162bdd8959f9406a56160c7ee78b82c37bf422614c194 WatchSource:0}: Error finding container 468f3cd7869773fa7f7162bdd8959f9406a56160c7ee78b82c37bf422614c194: Status 404 returned error can't find the container with id 468f3cd7869773fa7f7162bdd8959f9406a56160c7ee78b82c37bf422614c194
Jul 30 03:39:42.214630 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.214605    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:39:42.305710 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.305669    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") pod \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") "
Jul 30 03:39:42.305885 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.305728    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-trusted-ca\") pod \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") "
Jul 30 03:39:42.305885 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.305777    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-image-registry-private-configuration\") pod \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") "
Jul 30 03:39:42.305885 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.305825    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-bound-sa-token\") pod \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") "
Jul 30 03:39:42.305885 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.305857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-ca-trust-extracted\") pod \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") "
Jul 30 03:39:42.306195 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.306173    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-certificates\") pod \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") "
Jul 30 03:39:42.306300 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.306275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-installation-pull-secrets\") pod \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") "
Jul 30 03:39:42.306351 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.306316    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccqnj\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-kube-api-access-ccqnj\") pod \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\" (UID: \"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b\") "
Jul 30 03:39:42.306678 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.306641    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:39:42.306903 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.306843    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:39:42.308808 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.308704    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:39:42.308808 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.308712    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:39:42.309020 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.308984    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:39:42.309326 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.309303    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-kube-api-access-ccqnj" (OuterVolumeSpecName: "kube-api-access-ccqnj") pod "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b"). InnerVolumeSpecName "kube-api-access-ccqnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:39:42.309553 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.309533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:39:42.317178 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.317145    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" (UID: "a5d33d98-cae0-4ef5-94f2-e133d0a2d51b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:39:42.407758 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.407720    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-image-registry-private-configuration\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:39:42.407758 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.407757    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-bound-sa-token\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:39:42.407970 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.407773    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-ca-trust-extracted\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:39:42.407970 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.407788    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-certificates\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:39:42.407970 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.407802    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-installation-pull-secrets\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:39:42.407970 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.407815    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccqnj\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-kube-api-access-ccqnj\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:39:42.407970 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.407828    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-registry-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:39:42.407970 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.407840    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b-trusted-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:39:42.657541 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.657503    2560 generic.go:356] "Generic (PLEG): container finished" podID="a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" containerID="816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f" exitCode=0
Jul 30 03:39:42.657702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.657572    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f"
Jul 30 03:39:42.657702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.657587    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f" event={"ID":"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b","Type":"ContainerDied","Data":"816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f"}
Jul 30 03:39:42.657702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.657645    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65954c5f6d-hfc6f" event={"ID":"a5d33d98-cae0-4ef5-94f2-e133d0a2d51b","Type":"ContainerDied","Data":"105a9d6183c568896ee13fa54a812fa1ed52ea3c264ace3f64015b7347360665"}
Jul 30 03:39:42.657702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.657677    2560 scope.go:117] "RemoveContainer" containerID="816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f"
Jul 30 03:39:42.658890 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.658865    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh" event={"ID":"76f62d5f-1f8f-43b8-95d0-950a657878ef","Type":"ContainerStarted","Data":"468f3cd7869773fa7f7162bdd8959f9406a56160c7ee78b82c37bf422614c194"}
Jul 30 03:39:42.673580 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.673563    2560 scope.go:117] "RemoveContainer" containerID="816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f"
Jul 30 03:39:42.673878 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:42.673857    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f\": container with ID starting with 816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f not found: ID does not exist" containerID="816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f"
Jul 30 03:39:42.673925 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.673887    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f"} err="failed to get container status \"816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f\": rpc error: code = NotFound desc = could not find container \"816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f\": container with ID starting with 816478a54b5c2549a13590ef586715f24ed558e122da75dc5cfb08b932e1ee9f not found: ID does not exist"
Jul 30 03:39:42.711405 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.711379    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65954c5f6d-hfc6f"]
Jul 30 03:39:42.720156 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.720130    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65954c5f6d-hfc6f"]
Jul 30 03:39:42.727128 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.727097    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:39:42.727544 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.727517    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" containerName="registry"
Jul 30 03:39:42.727544 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.727539    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" containerName="registry"
Jul 30 03:39:42.727681 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.727619    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" containerName="registry"
Jul 30 03:39:42.731376 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.731355    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.735621 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.735602    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 30 03:39:42.735726 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.735605    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 30 03:39:42.735726 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.735609    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 30 03:39:42.737812 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.737796    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 30 03:39:42.741109 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.741090    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 30 03:39:42.741269 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.741251    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 30 03:39:42.742175 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.742157    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 30 03:39:42.742175 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.742161    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 30 03:39:42.742333 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.742216    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 30 03:39:42.742500 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.742466    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 30 03:39:42.742600 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.742522    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1d164bnkdhqgr\""
Jul 30 03:39:42.742600 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.742505    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 30 03:39:42.742600 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.742512    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ghfjc\""
Jul 30 03:39:42.742600 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.742574    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 30 03:39:42.745903 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.745883    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 30 03:39:42.758800 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.758594    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 30 03:39:42.765305 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.765278    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:39:42.811882 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.811851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.811882 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.811887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-web-config\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.811912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67sf\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-kube-api-access-w67sf\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.811977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-config\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812353    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812911 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812911 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-config-out\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812911 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.812911 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.812608    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913725 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913725 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913725 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913725 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913913 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-config-out\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913913 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913913 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913913 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913913 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-web-config\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.913913 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67sf\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-kube-api-access-w67sf\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.914274 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-config\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.914274 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.913976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.914274 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.914000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.914274 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.914038    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.914274 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.914062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.914274 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.914089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.914274 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.914125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.914626 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.914326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.915993 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.915200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.915993 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.915758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.916784 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.916552    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.916884 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.916819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.917586 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.917563    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.918059 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.918037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.918717 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.918676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.918918 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.918898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.919934 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.919204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.920609 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.920534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.921343 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.920703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-config-out\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.921343 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.921008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.921638 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.921581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-config\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.921638 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.921590    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.922054 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.922030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-web-config\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.922123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.922038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.922256 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.922182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.923605 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.923583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:42.928923 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:42.928903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67sf\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-kube-api-access-w67sf\") pod \"prometheus-k8s-0\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:43.042934 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:43.042862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:43.217192 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:43.216678    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:39:43.222052 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:39:43.222022    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode706c986_1009_4022_9f3f_f1abdf588a49.slice/crio-bf223d7769a04c491b5fdfd28ec15956a00d4dd03574984ff2e1af741b43dec7 WatchSource:0}: Error finding container bf223d7769a04c491b5fdfd28ec15956a00d4dd03574984ff2e1af741b43dec7: Status 404 returned error can't find the container with id bf223d7769a04c491b5fdfd28ec15956a00d4dd03574984ff2e1af741b43dec7
Jul 30 03:39:43.579596 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:43.579509    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d33d98-cae0-4ef5-94f2-e133d0a2d51b" path="/var/lib/kubelet/pods/a5d33d98-cae0-4ef5-94f2-e133d0a2d51b/volumes"
Jul 30 03:39:43.667688 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:43.667643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerStarted","Data":"bf223d7769a04c491b5fdfd28ec15956a00d4dd03574984ff2e1af741b43dec7"}
Jul 30 03:39:43.670097 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:43.670063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg" event={"ID":"45cc857f-c973-46f1-8cb3-7ae3efe45767","Type":"ContainerStarted","Data":"e4979e11bc5133bc6ee2cb1e88e997186261a1ed2a093551af609f2eea4e18ce"}
Jul 30 03:39:43.687023 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:43.686941    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg" podStartSLOduration=2.199119153 podStartE2EDuration="3.686920184s" totalImagesPullingTime="1.487801031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:43.686018571 +0000 UTC m=+290.640653283" watchObservedRunningTime="2026-07-30 03:39:43.686920184 +0000 UTC m=+290.641554856"
Jul 30 03:39:44.675862 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:44.675832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d" event={"ID":"f99ec3fe-a2c1-441a-8361-3f8f7fdcb886","Type":"ContainerStarted","Data":"4208294eb22c844ddb39072288bef510f290e6185458448b5b94b650266a7e6c"}
Jul 30 03:39:44.676194 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:44.676133    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:44.681163 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:44.681140    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d"
Jul 30 03:39:44.691195 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:44.691140    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kdm4d" podStartSLOduration=1.5318246229999999 podStartE2EDuration="3.691118087s" totalImagesPullingTime="2.159293464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:44.690035882 +0000 UTC m=+291.644670552" watchObservedRunningTime="2026-07-30 03:39:44.691118087 +0000 UTC m=+291.645752760"
Jul 30 03:39:45.681534 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:45.681496    2560 generic.go:356] "Generic (PLEG): container finished" podID="e706c986-1009-4022-9f3f-f1abdf588a49" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490" exitCode=0
Jul 30 03:39:45.681986 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:45.681583    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerDied","Data":"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"}
Jul 30 03:39:45.683922 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:45.683897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh" event={"ID":"76f62d5f-1f8f-43b8-95d0-950a657878ef","Type":"ContainerStarted","Data":"e6cc8b145d27079d1877e9b855e8c30e560582480129d8d7e7f7eadb25d98fa4"}
Jul 30 03:39:45.684003 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:45.683955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh" event={"ID":"76f62d5f-1f8f-43b8-95d0-950a657878ef","Type":"ContainerStarted","Data":"431a981b7bd77106051e8cfc1d88b393475317e4ae1f4ec009b5333e0a4b9e1d"}
Jul 30 03:39:45.684003 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:45.683980    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh" event={"ID":"76f62d5f-1f8f-43b8-95d0-950a657878ef","Type":"ContainerStarted","Data":"73505f7a454b646d836c81f476083f719b6fcfcf10ad009d643c8fc442888a5b"}
Jul 30 03:39:45.739462 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:45.739399    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5557d7d88f-dljgh" podStartSLOduration=2.220381644 podStartE2EDuration="4.739379384s" totalImagesPullingTime="2.51899774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:45.738458193 +0000 UTC m=+292.693092881" watchObservedRunningTime="2026-07-30 03:39:45.739379384 +0000 UTC m=+292.694014054"
Jul 30 03:39:47.307798 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:47.307750    2560 patch_prober.go:28] interesting pod/image-registry-5879b768bb-n7g4g container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 03:39:47.308198 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:47.307817    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g" podUID="d4565b6b-c6ff-474b-8208-b83400bce0c4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:39:49.572701 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:39:49.572657    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-pc47f" podUID="e5bd1a5c-9bd3-4874-991c-339874cf2ad0"
Jul 30 03:39:49.701739 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:49.701702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerStarted","Data":"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"}
Jul 30 03:39:49.701902 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:49.701770    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerStarted","Data":"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"}
Jul 30 03:39:50.543416 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:50.543389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5879b768bb-n7g4g"
Jul 30 03:39:51.661011 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:51.660979    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7t5ct"
Jul 30 03:39:51.714946 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:51.714912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerStarted","Data":"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"}
Jul 30 03:39:51.715086 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:51.714977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerStarted","Data":"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"}
Jul 30 03:39:51.715086 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:51.715007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerStarted","Data":"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"}
Jul 30 03:39:51.715086 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:51.715031    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerStarted","Data":"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"}
Jul 30 03:39:51.750732 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:51.750677    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.085158822 podStartE2EDuration="9.750653951s" totalImagesPullingTime="6.64283119s" totalInitContainerRuntime="22.663939ms" isStatefulPod=false podCreationTimestamp="2026-07-30 03:39:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:39:51.748924094 +0000 UTC m=+298.703558765" watchObservedRunningTime="2026-07-30 03:39:51.750653951 +0000 UTC m=+298.705288624"
Jul 30 03:39:53.043442 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:53.043404    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:39:53.442727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:39:53.442702    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 03:40:01.040447 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:01.040408    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:40:01.069953 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:01.040477    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:40:01.582860 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:01.582829    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:40:15.811918 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:15.811830    2560 generic.go:356] "Generic (PLEG): container finished" podID="62d54750-5010-4150-8945-09a630b12105" containerID="99571e3f0962a5af458d0809d9303d8931f19f5c27e007077092a8f4884e2bb6" exitCode=0
Jul 30 03:40:15.811918 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:15.811905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx" event={"ID":"62d54750-5010-4150-8945-09a630b12105","Type":"ContainerDied","Data":"99571e3f0962a5af458d0809d9303d8931f19f5c27e007077092a8f4884e2bb6"}
Jul 30 03:40:15.812386 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:15.812278    2560 scope.go:117] "RemoveContainer" containerID="99571e3f0962a5af458d0809d9303d8931f19f5c27e007077092a8f4884e2bb6"
Jul 30 03:40:16.817864 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:16.817833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-j56wx" event={"ID":"62d54750-5010-4150-8945-09a630b12105","Type":"ContainerStarted","Data":"ac7437f5184c694cddf1bf16a50db43406d6544868f59dfd9ba43f27a3ed3509"}
Jul 30 03:40:18.689688 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:18.689637    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ctdpt_58a0e78a-b220-4eb2-8435-7298c7e001f4/serve-healthcheck-canary/0.log"
Jul 30 03:40:21.045211 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:21.045178    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:40:21.049048 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:21.049029    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c57d6c8-bbgsg"
Jul 30 03:40:43.043501 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:43.043456    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:40:43.062814 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:43.062782    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:40:43.938550 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:40:43.938520    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:01.079499 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.079420    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:41:01.080041 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.079708    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="prometheus" containerID="cri-o://792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21" gracePeriod=600
Jul 30 03:41:01.080041 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.079749    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy-web" containerID="cri-o://44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607" gracePeriod=600
Jul 30 03:41:01.080041 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.079740    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="thanos-sidecar" containerID="cri-o://47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53" gracePeriod=600
Jul 30 03:41:01.080041 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.079743    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy" containerID="cri-o://d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342" gracePeriod=600
Jul 30 03:41:01.080041 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.079854    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="config-reloader" containerID="cri-o://5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4" gracePeriod=600
Jul 30 03:41:01.080041 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.079821    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901" gracePeriod=600
Jul 30 03:41:01.316322 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.316292    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:01.437906 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.437872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-grpc-tls\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438100 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.437913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-trusted-ca-bundle\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438100 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.437944    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-tls-assets\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438100 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.437964    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-config-out\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438100 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.437984    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-db\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438350 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438096    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-1\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438350 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438161    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-metrics-client-certs\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438350 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438189    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-tls\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438350 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438229    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-kube-rbac-proxy\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438350 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-web-config\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438350 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438330    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-2\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-config\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438403    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-serving-certs-ca-bundle\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438425    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438443    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w67sf\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-kube-api-access-w67sf\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438471    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-kubelet-serving-ca-bundle\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438510    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438536    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-0\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438564    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-thanos-prometheus-http-client-file\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.438635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438599    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-metrics-client-ca\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.439072 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e706c986-1009-4022-9f3f-f1abdf588a49\" (UID: \"e706c986-1009-4022-9f3f-f1abdf588a49\") "
Jul 30 03:41:01.439072 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438700    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:41:01.439072 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.438944    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.439072 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.439047    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:41:01.439296 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.439115    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.440792 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.440582    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:41:01.440792 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.440626    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:41:01.440792 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.440652    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.440792 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.440770    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-config-out" (OuterVolumeSpecName: "config-out") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:41:01.441131 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.440885    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.441131 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.441023    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:41:01.441290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.441270    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:41:01.441364 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.441307    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:41:01.441832 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.441706    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:41:01.441832 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.441719    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.441832 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.441797    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.442041 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.442015    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-kube-api-access-w67sf" (OuterVolumeSpecName: "kube-api-access-w67sf") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "kube-api-access-w67sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:41:01.442921 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.442888    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.443016 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.442931    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.443259 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.443211    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.443259 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.443247    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-config" (OuterVolumeSpecName: "config") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.453902 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.453879    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-web-config" (OuterVolumeSpecName: "web-config") pod "e706c986-1009-4022-9f3f-f1abdf588a49" (UID: "e706c986-1009-4022-9f3f-f1abdf588a49"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:41:01.540504 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540461    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-web-config\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540504 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540494    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540504 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540503    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-config\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540504 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540512    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540522    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67sf\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-kube-api-access-w67sf\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540531    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540540    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540549    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540557    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540565    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e706c986-1009-4022-9f3f-f1abdf588a49-configmap-metrics-client-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540573    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540582    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-grpc-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540590    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e706c986-1009-4022-9f3f-f1abdf588a49-tls-assets\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540599    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-config-out\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540606    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e706c986-1009-4022-9f3f-f1abdf588a49-prometheus-k8s-db\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540615    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-metrics-client-certs\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540623    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.540761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.540634    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e706c986-1009-4022-9f3f-f1abdf588a49-secret-kube-rbac-proxy\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:41:01.994368 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994336    2560 generic.go:356] "Generic (PLEG): container finished" podID="e706c986-1009-4022-9f3f-f1abdf588a49" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901" exitCode=0
Jul 30 03:41:01.994368 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994361    2560 generic.go:356] "Generic (PLEG): container finished" podID="e706c986-1009-4022-9f3f-f1abdf588a49" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342" exitCode=0
Jul 30 03:41:01.994368 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994367    2560 generic.go:356] "Generic (PLEG): container finished" podID="e706c986-1009-4022-9f3f-f1abdf588a49" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607" exitCode=0
Jul 30 03:41:01.994368 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994373    2560 generic.go:356] "Generic (PLEG): container finished" podID="e706c986-1009-4022-9f3f-f1abdf588a49" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53" exitCode=0
Jul 30 03:41:01.994368 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994378    2560 generic.go:356] "Generic (PLEG): container finished" podID="e706c986-1009-4022-9f3f-f1abdf588a49" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4" exitCode=0
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994383    2560 generic.go:356] "Generic (PLEG): container finished" podID="e706c986-1009-4022-9f3f-f1abdf588a49" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21" exitCode=0
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994414    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994418    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerDied","Data":"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"}
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerDied","Data":"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"}
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerDied","Data":"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"}
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994567    2560 scope.go:117] "RemoveContainer" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994583    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerDied","Data":"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"}
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994628    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerDied","Data":"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"}
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerDied","Data":"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"}
Jul 30 03:41:01.994763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:01.994714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e706c986-1009-4022-9f3f-f1abdf588a49","Type":"ContainerDied","Data":"bf223d7769a04c491b5fdfd28ec15956a00d4dd03574984ff2e1af741b43dec7"}
Jul 30 03:41:02.002388 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.002341    2560 scope.go:117] "RemoveContainer" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"
Jul 30 03:41:02.009306 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.009284    2560 scope.go:117] "RemoveContainer" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"
Jul 30 03:41:02.013986 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.013963    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:41:02.016704 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.016687    2560 scope.go:117] "RemoveContainer" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"
Jul 30 03:41:02.019928 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.019905    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:41:02.024038 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.024021    2560 scope.go:117] "RemoveContainer" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"
Jul 30 03:41:02.030832 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.030815    2560 scope.go:117] "RemoveContainer" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"
Jul 30 03:41:02.038304 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.038283    2560 scope.go:117] "RemoveContainer" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"
Jul 30 03:41:02.045121 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045085    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:41:02.045433 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045417    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="prometheus"
Jul 30 03:41:02.045433 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045433    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="prometheus"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045442    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="init-config-reloader"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045448    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="init-config-reloader"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045456    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy-thanos"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045462    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy-thanos"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045473    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy-web"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045478    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy-web"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045485    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="config-reloader"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045490    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="config-reloader"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045496    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="thanos-sidecar"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045502    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="thanos-sidecar"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045509    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045514    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045557    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy"
Jul 30 03:41:02.045558 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045566    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="thanos-sidecar"
Jul 30 03:41:02.046061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045572    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy-thanos"
Jul 30 03:41:02.046061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045578    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="kube-rbac-proxy-web"
Jul 30 03:41:02.046061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045585    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="prometheus"
Jul 30 03:41:02.046061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045592    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" containerName="config-reloader"
Jul 30 03:41:02.046061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.045693    2560 scope.go:117] "RemoveContainer" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"
Jul 30 03:41:02.046061 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:41:02.045988    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": container with ID starting with 2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901 not found: ID does not exist" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"
Jul 30 03:41:02.046061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.046013    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"} err="failed to get container status \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": rpc error: code = NotFound desc = could not find container \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": container with ID starting with 2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901 not found: ID does not exist"
Jul 30 03:41:02.046061 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.046042    2560 scope.go:117] "RemoveContainer" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"
Jul 30 03:41:02.046416 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:41:02.046310    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": container with ID starting with d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342 not found: ID does not exist" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"
Jul 30 03:41:02.046416 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.046350    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"} err="failed to get container status \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": rpc error: code = NotFound desc = could not find container \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": container with ID starting with d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342 not found: ID does not exist"
Jul 30 03:41:02.046416 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.046368    2560 scope.go:117] "RemoveContainer" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"
Jul 30 03:41:02.046625 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:41:02.046607    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": container with ID starting with 44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607 not found: ID does not exist" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"
Jul 30 03:41:02.046665 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.046630    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"} err="failed to get container status \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": rpc error: code = NotFound desc = could not find container \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": container with ID starting with 44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607 not found: ID does not exist"
Jul 30 03:41:02.046665 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.046645    2560 scope.go:117] "RemoveContainer" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"
Jul 30 03:41:02.046905 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:41:02.046889    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": container with ID starting with 47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53 not found: ID does not exist" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"
Jul 30 03:41:02.046950 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.046909    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"} err="failed to get container status \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": rpc error: code = NotFound desc = could not find container \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": container with ID starting with 47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53 not found: ID does not exist"
Jul 30 03:41:02.046950 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.046925    2560 scope.go:117] "RemoveContainer" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"
Jul 30 03:41:02.047170 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:41:02.047150    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": container with ID starting with 5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4 not found: ID does not exist" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"
Jul 30 03:41:02.047254 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.047178    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"} err="failed to get container status \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": rpc error: code = NotFound desc = could not find container \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": container with ID starting with 5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4 not found: ID does not exist"
Jul 30 03:41:02.047254 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.047200    2560 scope.go:117] "RemoveContainer" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"
Jul 30 03:41:02.047608 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:41:02.047585    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": container with ID starting with 792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21 not found: ID does not exist" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"
Jul 30 03:41:02.047667 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.047613    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"} err="failed to get container status \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": rpc error: code = NotFound desc = could not find container \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": container with ID starting with 792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21 not found: ID does not exist"
Jul 30 03:41:02.047667 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.047627    2560 scope.go:117] "RemoveContainer" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"
Jul 30 03:41:02.047861 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:41:02.047846    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": container with ID starting with 952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490 not found: ID does not exist" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"
Jul 30 03:41:02.047902 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.047864    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"} err="failed to get container status \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": rpc error: code = NotFound desc = could not find container \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": container with ID starting with 952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490 not found: ID does not exist"
Jul 30 03:41:02.047902 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.047876    2560 scope.go:117] "RemoveContainer" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"
Jul 30 03:41:02.048074 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048053    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"} err="failed to get container status \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": rpc error: code = NotFound desc = could not find container \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": container with ID starting with 2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901 not found: ID does not exist"
Jul 30 03:41:02.048113 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048075    2560 scope.go:117] "RemoveContainer" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"
Jul 30 03:41:02.048320 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048300    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"} err="failed to get container status \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": rpc error: code = NotFound desc = could not find container \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": container with ID starting with d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342 not found: ID does not exist"
Jul 30 03:41:02.048369 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048320    2560 scope.go:117] "RemoveContainer" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"
Jul 30 03:41:02.048547 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048533    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"} err="failed to get container status \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": rpc error: code = NotFound desc = could not find container \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": container with ID starting with 44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607 not found: ID does not exist"
Jul 30 03:41:02.048547 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048547    2560 scope.go:117] "RemoveContainer" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"
Jul 30 03:41:02.048723 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048709    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"} err="failed to get container status \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": rpc error: code = NotFound desc = could not find container \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": container with ID starting with 47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53 not found: ID does not exist"
Jul 30 03:41:02.048723 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048722    2560 scope.go:117] "RemoveContainer" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"
Jul 30 03:41:02.048930 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048914    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"} err="failed to get container status \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": rpc error: code = NotFound desc = could not find container \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": container with ID starting with 5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4 not found: ID does not exist"
Jul 30 03:41:02.048975 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.048930    2560 scope.go:117] "RemoveContainer" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"
Jul 30 03:41:02.049150 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.049129    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"} err="failed to get container status \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": rpc error: code = NotFound desc = could not find container \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": container with ID starting with 792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21 not found: ID does not exist"
Jul 30 03:41:02.049195 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.049151    2560 scope.go:117] "RemoveContainer" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"
Jul 30 03:41:02.049396 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.049378    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"} err="failed to get container status \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": rpc error: code = NotFound desc = could not find container \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": container with ID starting with 952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490 not found: ID does not exist"
Jul 30 03:41:02.049396 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.049396    2560 scope.go:117] "RemoveContainer" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"
Jul 30 03:41:02.049642 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.049624    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"} err="failed to get container status \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": rpc error: code = NotFound desc = could not find container \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": container with ID starting with 2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901 not found: ID does not exist"
Jul 30 03:41:02.049696 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.049643    2560 scope.go:117] "RemoveContainer" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"
Jul 30 03:41:02.049866 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.049847    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"} err="failed to get container status \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": rpc error: code = NotFound desc = could not find container \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": container with ID starting with d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342 not found: ID does not exist"
Jul 30 03:41:02.049908 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.049868    2560 scope.go:117] "RemoveContainer" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"
Jul 30 03:41:02.050091 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050074    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"} err="failed to get container status \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": rpc error: code = NotFound desc = could not find container \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": container with ID starting with 44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607 not found: ID does not exist"
Jul 30 03:41:02.050129 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050092    2560 scope.go:117] "RemoveContainer" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"
Jul 30 03:41:02.050341 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050321    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"} err="failed to get container status \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": rpc error: code = NotFound desc = could not find container \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": container with ID starting with 47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53 not found: ID does not exist"
Jul 30 03:41:02.050403 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050342    2560 scope.go:117] "RemoveContainer" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"
Jul 30 03:41:02.050533 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050518    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"} err="failed to get container status \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": rpc error: code = NotFound desc = could not find container \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": container with ID starting with 5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4 not found: ID does not exist"
Jul 30 03:41:02.050584 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050533    2560 scope.go:117] "RemoveContainer" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"
Jul 30 03:41:02.050765 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050745    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"} err="failed to get container status \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": rpc error: code = NotFound desc = could not find container \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": container with ID starting with 792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21 not found: ID does not exist"
Jul 30 03:41:02.050809 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050767    2560 scope.go:117] "RemoveContainer" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"
Jul 30 03:41:02.051016 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.050999    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"} err="failed to get container status \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": rpc error: code = NotFound desc = could not find container \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": container with ID starting with 952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490 not found: ID does not exist"
Jul 30 03:41:02.051016 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.051016    2560 scope.go:117] "RemoveContainer" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"
Jul 30 03:41:02.051106 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.051016    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.051450 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.051264    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"} err="failed to get container status \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": rpc error: code = NotFound desc = could not find container \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": container with ID starting with 2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901 not found: ID does not exist"
Jul 30 03:41:02.051450 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.051295    2560 scope.go:117] "RemoveContainer" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"
Jul 30 03:41:02.051723 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.051701    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"} err="failed to get container status \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": rpc error: code = NotFound desc = could not find container \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": container with ID starting with d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342 not found: ID does not exist"
Jul 30 03:41:02.051788 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.051723    2560 scope.go:117] "RemoveContainer" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"
Jul 30 03:41:02.051965 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.051947    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"} err="failed to get container status \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": rpc error: code = NotFound desc = could not find container \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": container with ID starting with 44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607 not found: ID does not exist"
Jul 30 03:41:02.052006 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.051967    2560 scope.go:117] "RemoveContainer" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"
Jul 30 03:41:02.052212 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.052191    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"} err="failed to get container status \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": rpc error: code = NotFound desc = could not find container \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": container with ID starting with 47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53 not found: ID does not exist"
Jul 30 03:41:02.052368 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.052214    2560 scope.go:117] "RemoveContainer" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"
Jul 30 03:41:02.052628 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.052609    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"} err="failed to get container status \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": rpc error: code = NotFound desc = could not find container \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": container with ID starting with 5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4 not found: ID does not exist"
Jul 30 03:41:02.052675 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.052629    2560 scope.go:117] "RemoveContainer" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"
Jul 30 03:41:02.052888 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.052864    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"} err="failed to get container status \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": rpc error: code = NotFound desc = could not find container \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": container with ID starting with 792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21 not found: ID does not exist"
Jul 30 03:41:02.052888 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.052882    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1d164bnkdhqgr\""
Jul 30 03:41:02.052972 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.052891    2560 scope.go:117] "RemoveContainer" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"
Jul 30 03:41:02.053210 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053192    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 30 03:41:02.053404 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053226    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"} err="failed to get container status \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": rpc error: code = NotFound desc = could not find container \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": container with ID starting with 952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490 not found: ID does not exist"
Jul 30 03:41:02.053404 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053271    2560 scope.go:117] "RemoveContainer" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"
Jul 30 03:41:02.053404 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053295    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 30 03:41:02.053404 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053305    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 30 03:41:02.053404 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053347    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 30 03:41:02.053404 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 30 03:41:02.053404 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 30 03:41:02.053736 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053260    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 30 03:41:02.053736 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053626    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 30 03:41:02.053736 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053646    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 30 03:41:02.053736 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053655    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ghfjc\""
Jul 30 03:41:02.053876 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053767    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"} err="failed to get container status \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": rpc error: code = NotFound desc = could not find container \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": container with ID starting with 2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901 not found: ID does not exist"
Jul 30 03:41:02.053876 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.053787    2560 scope.go:117] "RemoveContainer" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"
Jul 30 03:41:02.054102 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054078    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"} err="failed to get container status \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": rpc error: code = NotFound desc = could not find container \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": container with ID starting with d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342 not found: ID does not exist"
Jul 30 03:41:02.054182 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054103    2560 scope.go:117] "RemoveContainer" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"
Jul 30 03:41:02.054306 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054288    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 30 03:41:02.054366 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054307    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 30 03:41:02.054366 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054330    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 30 03:41:02.054556 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054537    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"} err="failed to get container status \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": rpc error: code = NotFound desc = could not find container \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": container with ID starting with 44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607 not found: ID does not exist"
Jul 30 03:41:02.054621 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054558    2560 scope.go:117] "RemoveContainer" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"
Jul 30 03:41:02.054802 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054779    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"} err="failed to get container status \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": rpc error: code = NotFound desc = could not find container \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": container with ID starting with 47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53 not found: ID does not exist"
Jul 30 03:41:02.054802 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.054802    2560 scope.go:117] "RemoveContainer" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"
Jul 30 03:41:02.055161 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.055101    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"} err="failed to get container status \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": rpc error: code = NotFound desc = could not find container \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": container with ID starting with 5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4 not found: ID does not exist"
Jul 30 03:41:02.055161 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.055130    2560 scope.go:117] "RemoveContainer" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"
Jul 30 03:41:02.055463 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.055431    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"} err="failed to get container status \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": rpc error: code = NotFound desc = could not find container \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": container with ID starting with 792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21 not found: ID does not exist"
Jul 30 03:41:02.055463 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.055459    2560 scope.go:117] "RemoveContainer" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"
Jul 30 03:41:02.055748 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.055705    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"} err="failed to get container status \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": rpc error: code = NotFound desc = could not find container \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": container with ID starting with 952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490 not found: ID does not exist"
Jul 30 03:41:02.055808 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.055750    2560 scope.go:117] "RemoveContainer" containerID="2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"
Jul 30 03:41:02.056171 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056148    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901"} err="failed to get container status \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": rpc error: code = NotFound desc = could not find container \"2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901\": container with ID starting with 2bfbd8946c784187ba43eec71483bdf246a2a58005fa706841fbec4f0fbf7901 not found: ID does not exist"
Jul 30 03:41:02.056171 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056170    2560 scope.go:117] "RemoveContainer" containerID="d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"
Jul 30 03:41:02.056514 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056495    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342"} err="failed to get container status \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": rpc error: code = NotFound desc = could not find container \"d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342\": container with ID starting with d45e20114b3ce654574edf9859b8e4401acfc94a91f4ae1aa6ba162046c89342 not found: ID does not exist"
Jul 30 03:41:02.056597 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056516    2560 scope.go:117] "RemoveContainer" containerID="44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"
Jul 30 03:41:02.056736 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056720    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 30 03:41:02.056819 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056761    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607"} err="failed to get container status \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": rpc error: code = NotFound desc = could not find container \"44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607\": container with ID starting with 44bd71abdeadf6c3baae87d7fb2bf59d86093ec7a3cf477107631bcd61485607 not found: ID does not exist"
Jul 30 03:41:02.056819 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056781    2560 scope.go:117] "RemoveContainer" containerID="47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"
Jul 30 03:41:02.056991 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056974    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53"} err="failed to get container status \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": rpc error: code = NotFound desc = could not find container \"47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53\": container with ID starting with 47d175b7e0ddfc224f5ecba6851e736d783bc9b5556553adec6021b7244b2c53 not found: ID does not exist"
Jul 30 03:41:02.057057 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.056992    2560 scope.go:117] "RemoveContainer" containerID="5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"
Jul 30 03:41:02.057285 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.057216    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4"} err="failed to get container status \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": rpc error: code = NotFound desc = could not find container \"5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4\": container with ID starting with 5d9e2ed3b3655f25d260640d2351996b17ff03fcff30fb7e05e62bca9d9d48b4 not found: ID does not exist"
Jul 30 03:41:02.057351 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.057301    2560 scope.go:117] "RemoveContainer" containerID="792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"
Jul 30 03:41:02.057569 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.057550    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21"} err="failed to get container status \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": rpc error: code = NotFound desc = could not find container \"792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21\": container with ID starting with 792912c0a85dee8ad33a46389789789f27ebc9b82df2613ca749dcf2bbe9eb21 not found: ID does not exist"
Jul 30 03:41:02.057569 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.057568    2560 scope.go:117] "RemoveContainer" containerID="952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"
Jul 30 03:41:02.057836 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.057810    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490"} err="failed to get container status \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": rpc error: code = NotFound desc = could not find container \"952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490\": container with ID starting with 952856a8e49dc9808c1d45c4e5381abdeb30af598356ac0c111bb6b12d981490 not found: ID does not exist"
Jul 30 03:41:02.058769 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.058753    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 30 03:41:02.065250 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.065211    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:41:02.146659 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.146659 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4mp\" (UniqueName: \"kubernetes.io/projected/f1683baa-26b5-4cde-a961-0404511036e1-kube-api-access-sf4mp\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.146997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-config\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1683baa-26b5-4cde-a961-0404511036e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147639 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1683baa-26b5-4cde-a961-0404511036e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147639 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.147639 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.147223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.248599 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-config\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.248599 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.248599 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.248865 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.248865 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1683baa-26b5-4cde-a961-0404511036e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.248865 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1683baa-26b5-4cde-a961-0404511036e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.248865 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.248865 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.249141 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.249141 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.249141 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.248971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.249141 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.249000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.249421 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.249158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.251951 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.251509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1683baa-26b5-4cde-a961-0404511036e1-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.251951 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.251851    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.251951 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.251857    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.251951 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.251887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.251951 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.251509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1683baa-26b5-4cde-a961-0404511036e1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.251951 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.251950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252408 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.251981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252408 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252408 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252408 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252408 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252408 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252108    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4mp\" (UniqueName: \"kubernetes.io/projected/f1683baa-26b5-4cde-a961-0404511036e1-kube-api-access-sf4mp\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252706 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252706 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252706 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.252941 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.252915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.253030 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.253004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.253593 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.253298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.253698 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.253606    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.253868 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.253847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-config\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.254062 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.254038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.254132 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.254076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1683baa-26b5-4cde-a961-0404511036e1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.254444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.254412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.254513 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.254470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.255104 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.255084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.255575 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.255556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1683baa-26b5-4cde-a961-0404511036e1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.261929 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.261908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4mp\" (UniqueName: \"kubernetes.io/projected/f1683baa-26b5-4cde-a961-0404511036e1-kube-api-access-sf4mp\") pod \"prometheus-k8s-0\" (UID: \"f1683baa-26b5-4cde-a961-0404511036e1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.362511 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.362471    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:02.504169 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:02.504095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 03:41:02.508044 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:41:02.508014    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1683baa_26b5_4cde_a961_0404511036e1.slice/crio-d351f082dd7aae94b9cb8a276021d9d070b51b49deb6176f2fcc829af23e30c0 WatchSource:0}: Error finding container d351f082dd7aae94b9cb8a276021d9d070b51b49deb6176f2fcc829af23e30c0: Status 404 returned error can't find the container with id d351f082dd7aae94b9cb8a276021d9d070b51b49deb6176f2fcc829af23e30c0
Jul 30 03:41:03.001194 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:03.001161    2560 generic.go:356] "Generic (PLEG): container finished" podID="f1683baa-26b5-4cde-a961-0404511036e1" containerID="4bc0044435e9445d9883307b419a235bca26116155c9088d4ffa36c47d4cd0d9" exitCode=0
Jul 30 03:41:03.001403 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:03.001256    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1683baa-26b5-4cde-a961-0404511036e1","Type":"ContainerDied","Data":"4bc0044435e9445d9883307b419a235bca26116155c9088d4ffa36c47d4cd0d9"}
Jul 30 03:41:03.001403 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:03.001317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1683baa-26b5-4cde-a961-0404511036e1","Type":"ContainerStarted","Data":"d351f082dd7aae94b9cb8a276021d9d070b51b49deb6176f2fcc829af23e30c0"}
Jul 30 03:41:03.577794 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:03.577704    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e706c986-1009-4022-9f3f-f1abdf588a49" path="/var/lib/kubelet/pods/e706c986-1009-4022-9f3f-f1abdf588a49/volumes"
Jul 30 03:41:04.010209 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:04.010176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1683baa-26b5-4cde-a961-0404511036e1","Type":"ContainerStarted","Data":"ceed0e0741ac8235cbc36557aed583c185658e3bd0c82da66edf1cfaba674018"}
Jul 30 03:41:04.010393 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:04.010268    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1683baa-26b5-4cde-a961-0404511036e1","Type":"ContainerStarted","Data":"7c64361bc546db9060541778154f91294b076e125eb0b086de774b24fa4a406a"}
Jul 30 03:41:04.010393 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:04.010302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1683baa-26b5-4cde-a961-0404511036e1","Type":"ContainerStarted","Data":"ae9d0ee4313efc22156fca7cfdcb73f5ac2d1033d162b3c96e8713eecec9e1ae"}
Jul 30 03:41:04.010393 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:04.010330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1683baa-26b5-4cde-a961-0404511036e1","Type":"ContainerStarted","Data":"0695947174e6469454829d04a3af506ec83b25c666ead42345529803785c2f51"}
Jul 30 03:41:04.010393 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:04.010358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1683baa-26b5-4cde-a961-0404511036e1","Type":"ContainerStarted","Data":"a71770338b596198473591257ae427770b84739ca0a62930d305daa65710c7be"}
Jul 30 03:41:04.010393 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:04.010382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1683baa-26b5-4cde-a961-0404511036e1","Type":"ContainerStarted","Data":"366222e89a6a59493490e9a0e019f53c75ba93e4585bb25902d03006b945ca6e"}
Jul 30 03:41:04.035982 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:04.035929    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.013718228 podStartE2EDuration="2.035912801s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.194573ms" isStatefulPod=false podCreationTimestamp="2026-07-30 03:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:41:04.034507985 +0000 UTC m=+370.989142654" watchObservedRunningTime="2026-07-30 03:41:04.035912801 +0000 UTC m=+370.990547472"
Jul 30 03:41:06.291397 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:06.291359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:41:06.293715 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:06.293686    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5bd1a5c-9bd3-4874-991c-339874cf2ad0-metrics-certs\") pod \"network-metrics-daemon-pc47f\" (UID: \"e5bd1a5c-9bd3-4874-991c-339874cf2ad0\") " pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:41:06.386128 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:06.386095    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bvtgp\""
Jul 30 03:41:06.394621 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:06.394594    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc47f"
Jul 30 03:41:06.525872 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:06.525836    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pc47f"]
Jul 30 03:41:06.530605 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:41:06.530576    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bd1a5c_9bd3_4874_991c_339874cf2ad0.slice/crio-9505d5ca17d5e7c6a8a8c8ff3e07209f416e19624f1ace421705d0d87549cc33 WatchSource:0}: Error finding container 9505d5ca17d5e7c6a8a8c8ff3e07209f416e19624f1ace421705d0d87549cc33: Status 404 returned error can't find the container with id 9505d5ca17d5e7c6a8a8c8ff3e07209f416e19624f1ace421705d0d87549cc33
Jul 30 03:41:06.532650 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:06.532634    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:41:07.025672 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:07.025629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pc47f" event={"ID":"e5bd1a5c-9bd3-4874-991c-339874cf2ad0","Type":"ContainerStarted","Data":"9505d5ca17d5e7c6a8a8c8ff3e07209f416e19624f1ace421705d0d87549cc33"}
Jul 30 03:41:07.363492 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:07.363408    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:41:08.031045 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:08.031005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pc47f" event={"ID":"e5bd1a5c-9bd3-4874-991c-339874cf2ad0","Type":"ContainerStarted","Data":"aa6d7a42ab0e00dadf7318ecd92310dbac93fedc69e02e3ff0858b2aa382249a"}
Jul 30 03:41:08.031269 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:08.031060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pc47f" event={"ID":"e5bd1a5c-9bd3-4874-991c-339874cf2ad0","Type":"ContainerStarted","Data":"6fbbc86b28df5135aaeb2747eda746a2845e1c84d23b9fb846720744ca1cc82e"}
Jul 30 03:41:08.047456 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:41:08.047407    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pc47f" podStartSLOduration=374.097011882 podStartE2EDuration="6m15.047392708s" totalImagesPullingTime="950.380826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:41:08.045281106 +0000 UTC m=+374.999915778" watchObservedRunningTime="2026-07-30 03:41:08.047392708 +0000 UTC m=+375.002027379"
Jul 30 03:42:02.362963 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:02.362923    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:42:02.378704 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:02.378675    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:42:03.270087 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:03.270056    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 03:42:42.686006 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.685967    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"]
Jul 30 03:42:42.689332 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.689306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"
Jul 30 03:42:42.691117 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.691085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:42:42.691117 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.691094    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 30 03:42:42.691417 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.691188    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zhqsr\""
Jul 30 03:42:42.702695 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.702665    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"]
Jul 30 03:42:42.837697 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.837659    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ca16d7c-efb9-4f48-b81c-2e5256771c7e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ttp2s\" (UID: \"9ca16d7c-efb9-4f48-b81c-2e5256771c7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"
Jul 30 03:42:42.837878 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.837719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrwp\" (UniqueName: \"kubernetes.io/projected/9ca16d7c-efb9-4f48-b81c-2e5256771c7e-kube-api-access-fsrwp\") pod \"openshift-lws-operator-fd8ccff4c-ttp2s\" (UID: \"9ca16d7c-efb9-4f48-b81c-2e5256771c7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"
Jul 30 03:42:42.939106 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.939017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrwp\" (UniqueName: \"kubernetes.io/projected/9ca16d7c-efb9-4f48-b81c-2e5256771c7e-kube-api-access-fsrwp\") pod \"openshift-lws-operator-fd8ccff4c-ttp2s\" (UID: \"9ca16d7c-efb9-4f48-b81c-2e5256771c7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"
Jul 30 03:42:42.939106 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.939095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ca16d7c-efb9-4f48-b81c-2e5256771c7e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ttp2s\" (UID: \"9ca16d7c-efb9-4f48-b81c-2e5256771c7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"
Jul 30 03:42:42.939458 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.939440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ca16d7c-efb9-4f48-b81c-2e5256771c7e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ttp2s\" (UID: \"9ca16d7c-efb9-4f48-b81c-2e5256771c7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"
Jul 30 03:42:42.947397 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:42.947361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrwp\" (UniqueName: \"kubernetes.io/projected/9ca16d7c-efb9-4f48-b81c-2e5256771c7e-kube-api-access-fsrwp\") pod \"openshift-lws-operator-fd8ccff4c-ttp2s\" (UID: \"9ca16d7c-efb9-4f48-b81c-2e5256771c7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"
Jul 30 03:42:43.006913 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:43.006870    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"
Jul 30 03:42:43.140210 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:43.140176    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s"]
Jul 30 03:42:43.143498 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:42:43.143465    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ca16d7c_efb9_4f48_b81c_2e5256771c7e.slice/crio-51aa961ade5d497a1323525bddcf524cebe3c235b2cb0f7b75afc8c10ae7d3dd WatchSource:0}: Error finding container 51aa961ade5d497a1323525bddcf524cebe3c235b2cb0f7b75afc8c10ae7d3dd: Status 404 returned error can't find the container with id 51aa961ade5d497a1323525bddcf524cebe3c235b2cb0f7b75afc8c10ae7d3dd
Jul 30 03:42:43.418896 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:43.418858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s" event={"ID":"9ca16d7c-efb9-4f48-b81c-2e5256771c7e","Type":"ContainerStarted","Data":"51aa961ade5d497a1323525bddcf524cebe3c235b2cb0f7b75afc8c10ae7d3dd"}
Jul 30 03:42:46.433652 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:46.433606    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s" event={"ID":"9ca16d7c-efb9-4f48-b81c-2e5256771c7e","Type":"ContainerStarted","Data":"8f75448e1281b3b66417090527e25dc9690326d3ea954d1bfb6e870c132e0f40"}
Jul 30 03:42:46.448693 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:42:46.448637    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ttp2s" podStartSLOduration=2.11426748 podStartE2EDuration="4.448619137s" totalImagesPullingTime="2.334351657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:42:46.447035096 +0000 UTC m=+473.401669766" watchObservedRunningTime="2026-07-30 03:42:46.448619137 +0000 UTC m=+473.403253817"
Jul 30 03:43:01.628176 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.628140    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"]
Jul 30 03:43:01.636181 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.636161    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.637865 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.637834    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 30 03:43:01.637995 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.637953    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rfxwq\""
Jul 30 03:43:01.638053 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.638007    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 30 03:43:01.638095 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.638079    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 30 03:43:01.638146 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.638131    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 30 03:43:01.645413 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.645389    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"]
Jul 30 03:43:01.807685 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.807651    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d585fc93-cf2b-4348-86b0-1b3ff8907f78-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.807861 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.807692    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d585fc93-cf2b-4348-86b0-1b3ff8907f78-tmp\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.807861 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.807720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d585fc93-cf2b-4348-86b0-1b3ff8907f78-manifests\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.807861 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.807844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d585fc93-cf2b-4348-86b0-1b3ff8907f78-webhook-cert\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.808016 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.807886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqr5x\" (UniqueName: \"kubernetes.io/projected/d585fc93-cf2b-4348-86b0-1b3ff8907f78-kube-api-access-rqr5x\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.908846 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.908811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d585fc93-cf2b-4348-86b0-1b3ff8907f78-webhook-cert\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.909032 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.908854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqr5x\" (UniqueName: \"kubernetes.io/projected/d585fc93-cf2b-4348-86b0-1b3ff8907f78-kube-api-access-rqr5x\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.909032 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.908887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d585fc93-cf2b-4348-86b0-1b3ff8907f78-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.909032 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.908920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d585fc93-cf2b-4348-86b0-1b3ff8907f78-tmp\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.909032 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.908954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d585fc93-cf2b-4348-86b0-1b3ff8907f78-manifests\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.909420 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.909396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d585fc93-cf2b-4348-86b0-1b3ff8907f78-manifests\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.911444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.911422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d585fc93-cf2b-4348-86b0-1b3ff8907f78-webhook-cert\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.911596 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.911578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d585fc93-cf2b-4348-86b0-1b3ff8907f78-tmp\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.911693 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.911678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d585fc93-cf2b-4348-86b0-1b3ff8907f78-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.923094 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.923070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqr5x\" (UniqueName: \"kubernetes.io/projected/d585fc93-cf2b-4348-86b0-1b3ff8907f78-kube-api-access-rqr5x\") pod \"opendatahub-operator-controller-manager-5946c6c8fb-xnvfc\" (UID: \"d585fc93-cf2b-4348-86b0-1b3ff8907f78\") " pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:01.948068 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:01.948043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:02.088058 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:02.088025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"]
Jul 30 03:43:02.091822 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:43:02.091793    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd585fc93_cf2b_4348_86b0_1b3ff8907f78.slice/crio-9b26fe26e04d48fcd8cb8d6ea2c40678cd35a5225e71c627b0f02866642c1bd9 WatchSource:0}: Error finding container 9b26fe26e04d48fcd8cb8d6ea2c40678cd35a5225e71c627b0f02866642c1bd9: Status 404 returned error can't find the container with id 9b26fe26e04d48fcd8cb8d6ea2c40678cd35a5225e71c627b0f02866642c1bd9
Jul 30 03:43:02.502622 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:02.502588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc" event={"ID":"d585fc93-cf2b-4348-86b0-1b3ff8907f78","Type":"ContainerStarted","Data":"9b26fe26e04d48fcd8cb8d6ea2c40678cd35a5225e71c627b0f02866642c1bd9"}
Jul 30 03:43:05.520098 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:05.520015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc" event={"ID":"d585fc93-cf2b-4348-86b0-1b3ff8907f78","Type":"ContainerStarted","Data":"43a67082a663756468fa42d9292403eed30335c29fa04bd685cba13972d90e9d"}
Jul 30 03:43:06.526500 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:06.526462    2560 generic.go:356] "Generic (PLEG): container finished" podID="d585fc93-cf2b-4348-86b0-1b3ff8907f78" containerID="43a67082a663756468fa42d9292403eed30335c29fa04bd685cba13972d90e9d" exitCode=0
Jul 30 03:43:06.526928 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:06.526506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc" event={"ID":"d585fc93-cf2b-4348-86b0-1b3ff8907f78","Type":"ContainerDied","Data":"43a67082a663756468fa42d9292403eed30335c29fa04bd685cba13972d90e9d"}
Jul 30 03:43:07.533575 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:07.533535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc" event={"ID":"d585fc93-cf2b-4348-86b0-1b3ff8907f78","Type":"ContainerStarted","Data":"e396b1dcdf6d9bd0aba1153dc1e4b2d3d96772e2546658f0ebe04d57072fa6d1"}
Jul 30 03:43:07.533990 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:07.533686    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:07.555033 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:07.554937    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc" podStartSLOduration=1.87880806 podStartE2EDuration="6.554917963s" totalImagesPullingTime="3.58403633s" totalInitContainerRuntime="1.092073573s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:43:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:43:07.554054655 +0000 UTC m=+494.508689330" watchObservedRunningTime="2026-07-30 03:43:07.554917963 +0000 UTC m=+494.509552634"
Jul 30 03:43:18.540074 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:18.540040    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5946c6c8fb-xnvfc"
Jul 30 03:43:30.361297 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.361227    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"]
Jul 30 03:43:30.369976 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.369941    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.372474 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.372451    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 30 03:43:30.372912 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.372888    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 30 03:43:30.373342 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.373320    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9mn9q\""
Jul 30 03:43:30.374095 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.374077    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 30 03:43:30.383816 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.383788    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"]
Jul 30 03:43:30.459313 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.459272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32d44a91-d63a-4e0c-837b-afff21107c7f-manager-config\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.459499 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.459328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d44a91-d63a-4e0c-837b-afff21107c7f-metrics-cert\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.459499 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.459350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32d44a91-d63a-4e0c-837b-afff21107c7f-cert\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.459499 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.459435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbqzl\" (UniqueName: \"kubernetes.io/projected/32d44a91-d63a-4e0c-837b-afff21107c7f-kube-api-access-wbqzl\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.560088 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.560047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32d44a91-d63a-4e0c-837b-afff21107c7f-manager-config\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.560310 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.560105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d44a91-d63a-4e0c-837b-afff21107c7f-metrics-cert\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.560310 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.560128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32d44a91-d63a-4e0c-837b-afff21107c7f-cert\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.560310 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.560163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbqzl\" (UniqueName: \"kubernetes.io/projected/32d44a91-d63a-4e0c-837b-afff21107c7f-kube-api-access-wbqzl\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.560694 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.560665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32d44a91-d63a-4e0c-837b-afff21107c7f-manager-config\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.562902 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.562869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32d44a91-d63a-4e0c-837b-afff21107c7f-cert\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.562999 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.562870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d44a91-d63a-4e0c-837b-afff21107c7f-metrics-cert\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.573196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.573172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbqzl\" (UniqueName: \"kubernetes.io/projected/32d44a91-d63a-4e0c-837b-afff21107c7f-kube-api-access-wbqzl\") pod \"lws-controller-manager-6bb6c44f89-hlv5b\" (UID: \"32d44a91-d63a-4e0c-837b-afff21107c7f\") " pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.684925 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.684892    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:30.818269 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:30.818225    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"]
Jul 30 03:43:30.820570 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:43:30.820543    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d44a91_d63a_4e0c_837b_afff21107c7f.slice/crio-67de311690e2e549bcfe4a0bbdfc51a410fddf34a3bad856b848ceacb7e90821 WatchSource:0}: Error finding container 67de311690e2e549bcfe4a0bbdfc51a410fddf34a3bad856b848ceacb7e90821: Status 404 returned error can't find the container with id 67de311690e2e549bcfe4a0bbdfc51a410fddf34a3bad856b848ceacb7e90821
Jul 30 03:43:31.639572 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.639532    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b" event={"ID":"32d44a91-d63a-4e0c-837b-afff21107c7f","Type":"ContainerStarted","Data":"67de311690e2e549bcfe4a0bbdfc51a410fddf34a3bad856b848ceacb7e90821"}
Jul 30 03:43:31.818880 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.818842    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"]
Jul 30 03:43:31.821776 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.821752    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:31.824174 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.824144    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-t68lr\""
Jul 30 03:43:31.824324 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.824307    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 30 03:43:31.838076 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.838045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"]
Jul 30 03:43:31.871881 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.871178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f36ccbe-8559-491d-840d-6ad6061a55dc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qwpsf\" (UID: \"5f36ccbe-8559-491d-840d-6ad6061a55dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:31.871881 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.871309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4xf\" (UniqueName: \"kubernetes.io/projected/5f36ccbe-8559-491d-840d-6ad6061a55dc-kube-api-access-cm4xf\") pod \"llmisvc-controller-manager-777c6bd7d5-qwpsf\" (UID: \"5f36ccbe-8559-491d-840d-6ad6061a55dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:31.973037 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.972941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f36ccbe-8559-491d-840d-6ad6061a55dc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qwpsf\" (UID: \"5f36ccbe-8559-491d-840d-6ad6061a55dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:31.973037 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.973005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4xf\" (UniqueName: \"kubernetes.io/projected/5f36ccbe-8559-491d-840d-6ad6061a55dc-kube-api-access-cm4xf\") pod \"llmisvc-controller-manager-777c6bd7d5-qwpsf\" (UID: \"5f36ccbe-8559-491d-840d-6ad6061a55dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:31.976330 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.976277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f36ccbe-8559-491d-840d-6ad6061a55dc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qwpsf\" (UID: \"5f36ccbe-8559-491d-840d-6ad6061a55dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:31.988247 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:31.988203    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4xf\" (UniqueName: \"kubernetes.io/projected/5f36ccbe-8559-491d-840d-6ad6061a55dc-kube-api-access-cm4xf\") pod \"llmisvc-controller-manager-777c6bd7d5-qwpsf\" (UID: \"5f36ccbe-8559-491d-840d-6ad6061a55dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:32.138254 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:32.138195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:32.494638 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:32.494613    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"]
Jul 30 03:43:32.496515 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:43:32.496490    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5f36ccbe_8559_491d_840d_6ad6061a55dc.slice/crio-dbfce5af9e3819eee21dcfc3695994588ad48bcf531e6b1aff72e8fcfc8b918c WatchSource:0}: Error finding container dbfce5af9e3819eee21dcfc3695994588ad48bcf531e6b1aff72e8fcfc8b918c: Status 404 returned error can't find the container with id dbfce5af9e3819eee21dcfc3695994588ad48bcf531e6b1aff72e8fcfc8b918c
Jul 30 03:43:32.645280 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:32.645224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf" event={"ID":"5f36ccbe-8559-491d-840d-6ad6061a55dc","Type":"ContainerStarted","Data":"dbfce5af9e3819eee21dcfc3695994588ad48bcf531e6b1aff72e8fcfc8b918c"}
Jul 30 03:43:32.646727 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:32.646704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b" event={"ID":"32d44a91-d63a-4e0c-837b-afff21107c7f","Type":"ContainerStarted","Data":"0b9b5d46a3cbc8c19d285fb3f2520522bdb84572f5baace3ae45211ce52a921b"}
Jul 30 03:43:32.646855 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:32.646757    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:43:35.663562 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:35.663521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf" event={"ID":"5f36ccbe-8559-491d-840d-6ad6061a55dc","Type":"ContainerStarted","Data":"b3ce9df56890acb01f35bf4c9886511e491e4d5c18c96b83ce00949401c7b15a"}
Jul 30 03:43:35.664116 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:35.663625    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:43:35.681159 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:35.681067    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf" podStartSLOduration=1.949935057 podStartE2EDuration="4.681034648s" totalImagesPullingTime="2.731099591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:43:35.680444191 +0000 UTC m=+522.635078866" watchObservedRunningTime="2026-07-30 03:43:35.681034648 +0000 UTC m=+522.635669318"
Jul 30 03:43:35.681459 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:35.681421    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b" podStartSLOduration=4.095932916 podStartE2EDuration="5.681406144s" totalImagesPullingTime="1.585473228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:43:32.666676616 +0000 UTC m=+519.621311288" watchObservedRunningTime="2026-07-30 03:43:35.681406144 +0000 UTC m=+522.636040816"
Jul 30 03:43:43.654095 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:43:43.654055    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bb6c44f89-hlv5b"
Jul 30 03:44:06.670230 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:06.670147    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qwpsf"
Jul 30 03:44:59.723702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.723665    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"]
Jul 30 03:44:59.727056 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.727035    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:44:59.728984 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.728961    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 30 03:44:59.729319 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.729301    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-b5q6p\""
Jul 30 03:44:59.729424 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.729321    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 30 03:44:59.743768 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.743738    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"]
Jul 30 03:44:59.781539 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.781502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxtj\" (UniqueName: \"kubernetes.io/projected/5b70d1f8-509b-4c38-8635-899e84f1986a-kube-api-access-kvxtj\") pod \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" (UID: \"5b70d1f8-509b-4c38-8635-899e84f1986a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:44:59.882498 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.882457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxtj\" (UniqueName: \"kubernetes.io/projected/5b70d1f8-509b-4c38-8635-899e84f1986a-kube-api-access-kvxtj\") pod \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" (UID: \"5b70d1f8-509b-4c38-8635-899e84f1986a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:44:59.889736 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:44:59.889710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxtj\" (UniqueName: \"kubernetes.io/projected/5b70d1f8-509b-4c38-8635-899e84f1986a-kube-api-access-kvxtj\") pod \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" (UID: \"5b70d1f8-509b-4c38-8635-899e84f1986a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:00.037034 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:00.036944    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:00.190929 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:00.190890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"]
Jul 30 03:45:00.195135 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:45:00.195098    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b70d1f8_509b_4c38_8635_899e84f1986a.slice/crio-bbc8c6168c3e9b227b9f24a0d67db6052925d5680f79807549fdc92fee8657c5 WatchSource:0}: Error finding container bbc8c6168c3e9b227b9f24a0d67db6052925d5680f79807549fdc92fee8657c5: Status 404 returned error can't find the container with id bbc8c6168c3e9b227b9f24a0d67db6052925d5680f79807549fdc92fee8657c5
Jul 30 03:45:01.048092 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:01.047985    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6" event={"ID":"5b70d1f8-509b-4c38-8635-899e84f1986a","Type":"ContainerStarted","Data":"bbc8c6168c3e9b227b9f24a0d67db6052925d5680f79807549fdc92fee8657c5"}
Jul 30 03:45:03.059398 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:03.059360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6" event={"ID":"5b70d1f8-509b-4c38-8635-899e84f1986a","Type":"ContainerStarted","Data":"c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e"}
Jul 30 03:45:03.059801 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:03.059485    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:03.074350 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:03.074294    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6" podStartSLOduration=1.792585415 podStartE2EDuration="4.074274897s" totalImagesPullingTime="2.281689482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:03.073408982 +0000 UTC m=+610.028043654" watchObservedRunningTime="2026-07-30 03:45:03.074274897 +0000 UTC m=+610.028909568"
Jul 30 03:45:12.236443 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.236405    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"]
Jul 30 03:45:12.236963 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.236684    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" containerName="manager" containerID="cri-o://c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e" gracePeriod=2
Jul 30 03:45:12.238666 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.238639    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:12.241829 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.241795    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"]
Jul 30 03:45:12.243202 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:12.243068    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" is forbidden: User \"system:node:ip-10-0-137-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-125.ec2.internal' and this object" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:12.260329 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.260300    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"]
Jul 30 03:45:12.260767 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.260744    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" containerName="manager"
Jul 30 03:45:12.260767 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.260768    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" containerName="manager"
Jul 30 03:45:12.260939 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.260848    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" containerName="manager"
Jul 30 03:45:12.264084 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.264066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"
Jul 30 03:45:12.276646 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.276617    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"]
Jul 30 03:45:12.295219 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:12.295174    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" is forbidden: User \"system:node:ip-10-0-137-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-125.ec2.internal' and this object" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:12.392159 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.392125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm98f\" (UniqueName: \"kubernetes.io/projected/6f3a62dd-136a-4929-8670-d4b7a90a1a43-kube-api-access-jm98f\") pod \"limitador-operator-controller-manager-859d7b78bd-957ss\" (UID: \"6f3a62dd-136a-4929-8670-d4b7a90a1a43\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"
Jul 30 03:45:12.474517 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.474492    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:12.476173 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:12.476144    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" is forbidden: User \"system:node:ip-10-0-137-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-125.ec2.internal' and this object" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:12.493638 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.493569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm98f\" (UniqueName: \"kubernetes.io/projected/6f3a62dd-136a-4929-8670-d4b7a90a1a43-kube-api-access-jm98f\") pod \"limitador-operator-controller-manager-859d7b78bd-957ss\" (UID: \"6f3a62dd-136a-4929-8670-d4b7a90a1a43\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"
Jul 30 03:45:12.501459 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.501433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm98f\" (UniqueName: \"kubernetes.io/projected/6f3a62dd-136a-4929-8670-d4b7a90a1a43-kube-api-access-jm98f\") pod \"limitador-operator-controller-manager-859d7b78bd-957ss\" (UID: \"6f3a62dd-136a-4929-8670-d4b7a90a1a43\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"
Jul 30 03:45:12.594294 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.594259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvxtj\" (UniqueName: \"kubernetes.io/projected/5b70d1f8-509b-4c38-8635-899e84f1986a-kube-api-access-kvxtj\") pod \"5b70d1f8-509b-4c38-8635-899e84f1986a\" (UID: \"5b70d1f8-509b-4c38-8635-899e84f1986a\") "
Jul 30 03:45:12.596434 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.596406    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b70d1f8-509b-4c38-8635-899e84f1986a-kube-api-access-kvxtj" (OuterVolumeSpecName: "kube-api-access-kvxtj") pod "5b70d1f8-509b-4c38-8635-899e84f1986a" (UID: "5b70d1f8-509b-4c38-8635-899e84f1986a"). InnerVolumeSpecName "kube-api-access-kvxtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:45:12.607530 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.607506    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"
Jul 30 03:45:12.695948 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.695912    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxtj\" (UniqueName: \"kubernetes.io/projected/5b70d1f8-509b-4c38-8635-899e84f1986a-kube-api-access-kvxtj\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:45:12.741913 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:12.741759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"]
Jul 30 03:45:12.744739 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:45:12.744677    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3a62dd_136a_4929_8670_d4b7a90a1a43.slice/crio-e3fa4b1acdaea8748e853a20230b77b6547bd0f4f02d3b02ed7565931cc14a80 WatchSource:0}: Error finding container e3fa4b1acdaea8748e853a20230b77b6547bd0f4f02d3b02ed7565931cc14a80: Status 404 returned error can't find the container with id e3fa4b1acdaea8748e853a20230b77b6547bd0f4f02d3b02ed7565931cc14a80
Jul 30 03:45:13.103919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.103829    2560 generic.go:356] "Generic (PLEG): container finished" podID="5b70d1f8-509b-4c38-8635-899e84f1986a" containerID="c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e" exitCode=0
Jul 30 03:45:13.103919 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.103881    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:13.104123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.103931    2560 scope.go:117] "RemoveContainer" containerID="c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e"
Jul 30 03:45:13.105516 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.105487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss" event={"ID":"6f3a62dd-136a-4929-8670-d4b7a90a1a43","Type":"ContainerStarted","Data":"8a6268583ebd409107bee2e1630fb9e045cf98016419f0ee9d12e72f06a20f9e"}
Jul 30 03:45:13.105636 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.105534    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss" event={"ID":"6f3a62dd-136a-4929-8670-d4b7a90a1a43","Type":"ContainerStarted","Data":"e3fa4b1acdaea8748e853a20230b77b6547bd0f4f02d3b02ed7565931cc14a80"}
Jul 30 03:45:13.105705 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.105635    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"
Jul 30 03:45:13.106901 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:13.106876    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" is forbidden: User \"system:node:ip-10-0-137-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-125.ec2.internal' and this object" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:13.113039 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.113022    2560 scope.go:117] "RemoveContainer" containerID="c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e"
Jul 30 03:45:13.113364 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:13.113344    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e\": container with ID starting with c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e not found: ID does not exist" containerID="c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e"
Jul 30 03:45:13.113419 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.113372    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e"} err="failed to get container status \"c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e\": rpc error: code = NotFound desc = could not find container \"c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e\": container with ID starting with c2d92901ce893af15b9209195bcf2aedb91a3e13dd43f0a7e7ae59385ab35b8e not found: ID does not exist"
Jul 30 03:45:13.126023 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:13.125993    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" is forbidden: User \"system:node:ip-10-0-137-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-125.ec2.internal' and this object" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:13.126150 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.126005    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss" podStartSLOduration=1.1259905510000001 podStartE2EDuration="1.125990551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:13.124212934 +0000 UTC m=+620.078847606" watchObservedRunningTime="2026-07-30 03:45:13.125990551 +0000 UTC m=+620.080625221"
Jul 30 03:45:13.127222 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:13.127201    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" is forbidden: User \"system:node:ip-10-0-137-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-125.ec2.internal' and this object" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:13.220578 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.220543    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"]
Jul 30 03:45:13.225027 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.225009    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:13.226639 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:13.226607    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" is forbidden: User \"system:node:ip-10-0-137-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-125.ec2.internal' and this object" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:13.226875 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.226857    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6v4xr\""
Jul 30 03:45:13.238544 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.238518    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"]
Jul 30 03:45:13.400406 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.400367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412bb0ea-9f30-4804-a9ef-b805994ae193-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kpt2c\" (UID: \"412bb0ea-9f30-4804-a9ef-b805994ae193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:13.400553 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.400520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmc7\" (UniqueName: \"kubernetes.io/projected/412bb0ea-9f30-4804-a9ef-b805994ae193-kube-api-access-pdmc7\") pod \"kuadrant-operator-controller-manager-86878c769f-kpt2c\" (UID: \"412bb0ea-9f30-4804-a9ef-b805994ae193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:13.501081 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.501045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmc7\" (UniqueName: \"kubernetes.io/projected/412bb0ea-9f30-4804-a9ef-b805994ae193-kube-api-access-pdmc7\") pod \"kuadrant-operator-controller-manager-86878c769f-kpt2c\" (UID: \"412bb0ea-9f30-4804-a9ef-b805994ae193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:13.501283 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.501092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412bb0ea-9f30-4804-a9ef-b805994ae193-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kpt2c\" (UID: \"412bb0ea-9f30-4804-a9ef-b805994ae193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:13.501522 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.501502    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412bb0ea-9f30-4804-a9ef-b805994ae193-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kpt2c\" (UID: \"412bb0ea-9f30-4804-a9ef-b805994ae193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:13.512775 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.512753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmc7\" (UniqueName: \"kubernetes.io/projected/412bb0ea-9f30-4804-a9ef-b805994ae193-kube-api-access-pdmc7\") pod \"kuadrant-operator-controller-manager-86878c769f-kpt2c\" (UID: \"412bb0ea-9f30-4804-a9ef-b805994ae193\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:13.535196 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.535156    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:13.576679 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:45:13.576641    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hr9p6\" is forbidden: User \"system:node:ip-10-0-137-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-125.ec2.internal' and this object" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hr9p6"
Jul 30 03:45:13.578837 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.578811    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b70d1f8-509b-4c38-8635-899e84f1986a" path="/var/lib/kubelet/pods/5b70d1f8-509b-4c38-8635-899e84f1986a/volumes"
Jul 30 03:45:13.678297 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:13.678267    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"]
Jul 30 03:45:13.680965 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:45:13.680937    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412bb0ea_9f30_4804_a9ef_b805994ae193.slice/crio-6955443792ed18a2f82cd25e19bbf0a1b272e08b6b82d77b2407c64cf96f04b3 WatchSource:0}: Error finding container 6955443792ed18a2f82cd25e19bbf0a1b272e08b6b82d77b2407c64cf96f04b3: Status 404 returned error can't find the container with id 6955443792ed18a2f82cd25e19bbf0a1b272e08b6b82d77b2407c64cf96f04b3
Jul 30 03:45:14.113393 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:14.113308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c" event={"ID":"412bb0ea-9f30-4804-a9ef-b805994ae193","Type":"ContainerStarted","Data":"6955443792ed18a2f82cd25e19bbf0a1b272e08b6b82d77b2407c64cf96f04b3"}
Jul 30 03:45:18.133391 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:18.133359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c" event={"ID":"412bb0ea-9f30-4804-a9ef-b805994ae193","Type":"ContainerStarted","Data":"3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a"}
Jul 30 03:45:18.133792 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:18.133489    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:45:18.149755 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:18.149710    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c" podStartSLOduration=0.802316491 podStartE2EDuration="5.149696047s" totalImagesPullingTime="4.347379556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:18.149353727 +0000 UTC m=+625.103988399" watchObservedRunningTime="2026-07-30 03:45:18.149696047 +0000 UTC m=+625.104330717"
Jul 30 03:45:24.116388 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:24.116359    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-957ss"
Jul 30 03:45:29.141075 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:45:29.141046    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:46:27.578024 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.577990    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77fbb8d78d-l9lzc"]
Jul 30 03:46:27.581478 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.581462    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:27.583553 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.583517    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 03:46:27.583667 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.583609    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-674r7\""
Jul 30 03:46:27.583899 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.583884    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 03:46:27.593009 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.592964    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77fbb8d78d-l9lzc"]
Jul 30 03:46:27.673875 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.673836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12637552-350e-4e6b-9ac6-f2c8d176a675-webhook-cert\") pod \"maas-controller-77fbb8d78d-l9lzc\" (UID: \"12637552-350e-4e6b-9ac6-f2c8d176a675\") " pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:27.674049 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.673919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2264\" (UniqueName: \"kubernetes.io/projected/12637552-350e-4e6b-9ac6-f2c8d176a675-kube-api-access-p2264\") pod \"maas-controller-77fbb8d78d-l9lzc\" (UID: \"12637552-350e-4e6b-9ac6-f2c8d176a675\") " pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:27.709962 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.709931    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-8b7vq"]
Jul 30 03:46:27.713684 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.713663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:27.721011 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.720976    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-8b7vq"]
Jul 30 03:46:27.775247 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.775201    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484wp\" (UniqueName: \"kubernetes.io/projected/e1f51f45-286b-4591-8758-05f95104d4c2-kube-api-access-484wp\") pod \"maas-controller-5b9bc959cf-8b7vq\" (UID: \"e1f51f45-286b-4591-8758-05f95104d4c2\") " pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:27.775420 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.775292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12637552-350e-4e6b-9ac6-f2c8d176a675-webhook-cert\") pod \"maas-controller-77fbb8d78d-l9lzc\" (UID: \"12637552-350e-4e6b-9ac6-f2c8d176a675\") " pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:27.775420 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.775313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1f51f45-286b-4591-8758-05f95104d4c2-webhook-cert\") pod \"maas-controller-5b9bc959cf-8b7vq\" (UID: \"e1f51f45-286b-4591-8758-05f95104d4c2\") " pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:27.775420 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.775347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2264\" (UniqueName: \"kubernetes.io/projected/12637552-350e-4e6b-9ac6-f2c8d176a675-kube-api-access-p2264\") pod \"maas-controller-77fbb8d78d-l9lzc\" (UID: \"12637552-350e-4e6b-9ac6-f2c8d176a675\") " pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:27.777940 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.777906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12637552-350e-4e6b-9ac6-f2c8d176a675-webhook-cert\") pod \"maas-controller-77fbb8d78d-l9lzc\" (UID: \"12637552-350e-4e6b-9ac6-f2c8d176a675\") " pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:27.782751 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.782721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2264\" (UniqueName: \"kubernetes.io/projected/12637552-350e-4e6b-9ac6-f2c8d176a675-kube-api-access-p2264\") pod \"maas-controller-77fbb8d78d-l9lzc\" (UID: \"12637552-350e-4e6b-9ac6-f2c8d176a675\") " pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:27.876380 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.876274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484wp\" (UniqueName: \"kubernetes.io/projected/e1f51f45-286b-4591-8758-05f95104d4c2-kube-api-access-484wp\") pod \"maas-controller-5b9bc959cf-8b7vq\" (UID: \"e1f51f45-286b-4591-8758-05f95104d4c2\") " pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:27.876380 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.876371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1f51f45-286b-4591-8758-05f95104d4c2-webhook-cert\") pod \"maas-controller-5b9bc959cf-8b7vq\" (UID: \"e1f51f45-286b-4591-8758-05f95104d4c2\") " pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:27.878941 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.878908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1f51f45-286b-4591-8758-05f95104d4c2-webhook-cert\") pod \"maas-controller-5b9bc959cf-8b7vq\" (UID: \"e1f51f45-286b-4591-8758-05f95104d4c2\") " pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:27.883986 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.883957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-484wp\" (UniqueName: \"kubernetes.io/projected/e1f51f45-286b-4591-8758-05f95104d4c2-kube-api-access-484wp\") pod \"maas-controller-5b9bc959cf-8b7vq\" (UID: \"e1f51f45-286b-4591-8758-05f95104d4c2\") " pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:27.894215 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:27.894191    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:28.023153 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:28.023047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77fbb8d78d-l9lzc"]
Jul 30 03:46:28.025500 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:28.025476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:28.025813 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:46:28.025791    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12637552_350e_4e6b_9ac6_f2c8d176a675.slice/crio-5a72c91c642e020561fd9cdd3c0397702505aa6bf1519325aee7f63ce8805613 WatchSource:0}: Error finding container 5a72c91c642e020561fd9cdd3c0397702505aa6bf1519325aee7f63ce8805613: Status 404 returned error can't find the container with id 5a72c91c642e020561fd9cdd3c0397702505aa6bf1519325aee7f63ce8805613
Jul 30 03:46:28.027093 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:28.027079    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:46:28.154446 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:28.154317    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-8b7vq"]
Jul 30 03:46:28.157044 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:46:28.157015    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f51f45_286b_4591_8758_05f95104d4c2.slice/crio-fb1fe4157520880ed0843255faa79259597f85c9049118f26e57f69610942914 WatchSource:0}: Error finding container fb1fe4157520880ed0843255faa79259597f85c9049118f26e57f69610942914: Status 404 returned error can't find the container with id fb1fe4157520880ed0843255faa79259597f85c9049118f26e57f69610942914
Jul 30 03:46:28.453349 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:28.453262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc" event={"ID":"12637552-350e-4e6b-9ac6-f2c8d176a675","Type":"ContainerStarted","Data":"5a72c91c642e020561fd9cdd3c0397702505aa6bf1519325aee7f63ce8805613"}
Jul 30 03:46:28.454351 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:28.454329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq" event={"ID":"e1f51f45-286b-4591-8758-05f95104d4c2","Type":"ContainerStarted","Data":"fb1fe4157520880ed0843255faa79259597f85c9049118f26e57f69610942914"}
Jul 30 03:46:31.474595 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:31.474559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq" event={"ID":"e1f51f45-286b-4591-8758-05f95104d4c2","Type":"ContainerStarted","Data":"66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59"}
Jul 30 03:46:31.475035 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:31.474650    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:31.476104 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:31.476080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc" event={"ID":"12637552-350e-4e6b-9ac6-f2c8d176a675","Type":"ContainerStarted","Data":"b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf"}
Jul 30 03:46:31.476155 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:31.476136    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:31.492800 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:31.492740    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq" podStartSLOduration=2.014208997 podStartE2EDuration="4.492722978s" totalImagesPullingTime="2.478513981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:31.490995764 +0000 UTC m=+698.445630457" watchObservedRunningTime="2026-07-30 03:46:31.492722978 +0000 UTC m=+698.447357649"
Jul 30 03:46:31.513554 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:31.513506    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc" podStartSLOduration=1.896978831 podStartE2EDuration="4.513486856s" totalImagesPullingTime="2.616508025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:31.512576327 +0000 UTC m=+698.467210998" watchObservedRunningTime="2026-07-30 03:46:31.513486856 +0000 UTC m=+698.468121527"
Jul 30 03:46:42.482655 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.482625    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:46:42.483163 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.482707    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:42.541359 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.541314    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77fbb8d78d-l9lzc"]
Jul 30 03:46:42.541574 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.541539    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc" podUID="12637552-350e-4e6b-9ac6-f2c8d176a675" containerName="manager" containerID="cri-o://b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf" gracePeriod=30
Jul 30 03:46:42.789916 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.789889    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:42.912296 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.912257    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2264\" (UniqueName: \"kubernetes.io/projected/12637552-350e-4e6b-9ac6-f2c8d176a675-kube-api-access-p2264\") pod \"12637552-350e-4e6b-9ac6-f2c8d176a675\" (UID: \"12637552-350e-4e6b-9ac6-f2c8d176a675\") "
Jul 30 03:46:42.912464 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.912328    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12637552-350e-4e6b-9ac6-f2c8d176a675-webhook-cert\") pod \"12637552-350e-4e6b-9ac6-f2c8d176a675\" (UID: \"12637552-350e-4e6b-9ac6-f2c8d176a675\") "
Jul 30 03:46:42.914444 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.914417    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12637552-350e-4e6b-9ac6-f2c8d176a675-kube-api-access-p2264" (OuterVolumeSpecName: "kube-api-access-p2264") pod "12637552-350e-4e6b-9ac6-f2c8d176a675" (UID: "12637552-350e-4e6b-9ac6-f2c8d176a675"). InnerVolumeSpecName "kube-api-access-p2264". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:46:42.914556 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:42.914532    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12637552-350e-4e6b-9ac6-f2c8d176a675-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "12637552-350e-4e6b-9ac6-f2c8d176a675" (UID: "12637552-350e-4e6b-9ac6-f2c8d176a675"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:46:43.013167 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.013126    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2264\" (UniqueName: \"kubernetes.io/projected/12637552-350e-4e6b-9ac6-f2c8d176a675-kube-api-access-p2264\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:46:43.013167 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.013163    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12637552-350e-4e6b-9ac6-f2c8d176a675-webhook-cert\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:46:43.533105 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.533068    2560 generic.go:356] "Generic (PLEG): container finished" podID="12637552-350e-4e6b-9ac6-f2c8d176a675" containerID="b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf" exitCode=0
Jul 30 03:46:43.533511 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.533122    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc"
Jul 30 03:46:43.533511 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.533167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc" event={"ID":"12637552-350e-4e6b-9ac6-f2c8d176a675","Type":"ContainerDied","Data":"b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf"}
Jul 30 03:46:43.533511 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.533227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77fbb8d78d-l9lzc" event={"ID":"12637552-350e-4e6b-9ac6-f2c8d176a675","Type":"ContainerDied","Data":"5a72c91c642e020561fd9cdd3c0397702505aa6bf1519325aee7f63ce8805613"}
Jul 30 03:46:43.533511 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.533270    2560 scope.go:117] "RemoveContainer" containerID="b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf"
Jul 30 03:46:43.542353 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.542177    2560 scope.go:117] "RemoveContainer" containerID="b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf"
Jul 30 03:46:43.542512 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:46:43.542495    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf\": container with ID starting with b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf not found: ID does not exist" containerID="b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf"
Jul 30 03:46:43.542557 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.542520    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf"} err="failed to get container status \"b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf\": rpc error: code = NotFound desc = could not find container \"b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf\": container with ID starting with b65db452fb6de1fe2c74a34bc9fb271e0e3bf16f7ad2fe56d3bff49cf305c0cf not found: ID does not exist"
Jul 30 03:46:43.551918 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.551890    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77fbb8d78d-l9lzc"]
Jul 30 03:46:43.555509 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.555488    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77fbb8d78d-l9lzc"]
Jul 30 03:46:43.578775 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:46:43.578747    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12637552-350e-4e6b-9ac6-f2c8d176a675" path="/var/lib/kubelet/pods/12637552-350e-4e6b-9ac6-f2c8d176a675/volumes"
Jul 30 03:47:05.107614 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.107571    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"]
Jul 30 03:47:05.108124 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.108109    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12637552-350e-4e6b-9ac6-f2c8d176a675" containerName="manager"
Jul 30 03:47:05.108178 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.108127    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="12637552-350e-4e6b-9ac6-f2c8d176a675" containerName="manager"
Jul 30 03:47:05.108260 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.108248    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="12637552-350e-4e6b-9ac6-f2c8d176a675" containerName="manager"
Jul 30 03:47:05.116979 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.116956    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.119254 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.119217    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-76fjd\""
Jul 30 03:47:05.119438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.119305    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 30 03:47:05.119646 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.119375    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 03:47:05.119724 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.119378    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 30 03:47:05.121066 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.121044    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"]
Jul 30 03:47:05.202674 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.202631    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a7f08d-1113-4c31-837f-0db8f106a7dd-openshift-service-ca\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.202674 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.202666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7f08d-1113-4c31-837f-0db8f106a7dd-maas-api-tls\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.202923 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.202786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnc5m\" (UniqueName: \"kubernetes.io/projected/20a7f08d-1113-4c31-837f-0db8f106a7dd-kube-api-access-pnc5m\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.303722 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.303686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a7f08d-1113-4c31-837f-0db8f106a7dd-openshift-service-ca\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.303910 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.303735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7f08d-1113-4c31-837f-0db8f106a7dd-maas-api-tls\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.303910 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.303852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnc5m\" (UniqueName: \"kubernetes.io/projected/20a7f08d-1113-4c31-837f-0db8f106a7dd-kube-api-access-pnc5m\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.304466 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.304437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a7f08d-1113-4c31-837f-0db8f106a7dd-openshift-service-ca\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.306957 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.306932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7f08d-1113-4c31-837f-0db8f106a7dd-maas-api-tls\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.310951 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.310926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnc5m\" (UniqueName: \"kubernetes.io/projected/20a7f08d-1113-4c31-837f-0db8f106a7dd-kube-api-access-pnc5m\") pod \"maas-api-95d5b8d55-vknqp\" (UID: \"20a7f08d-1113-4c31-837f-0db8f106a7dd\") " pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.429358 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.429323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:05.579507 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:47:05.579479    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a7f08d_1113_4c31_837f_0db8f106a7dd.slice/crio-de9769f816d9d3c31b1d3c48e0619897da96c70978cb59a47f5631083a7014a2 WatchSource:0}: Error finding container de9769f816d9d3c31b1d3c48e0619897da96c70978cb59a47f5631083a7014a2: Status 404 returned error can't find the container with id de9769f816d9d3c31b1d3c48e0619897da96c70978cb59a47f5631083a7014a2
Jul 30 03:47:05.580290 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.580268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"]
Jul 30 03:47:05.637716 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.637677    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp" event={"ID":"20a7f08d-1113-4c31-837f-0db8f106a7dd","Type":"ContainerStarted","Data":"de9769f816d9d3c31b1d3c48e0619897da96c70978cb59a47f5631083a7014a2"}
Jul 30 03:47:05.647417 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.647384    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"]
Jul 30 03:47:05.651939 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.651921    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:05.654301 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.654280    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 30 03:47:05.654396 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.654359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vb6c4\""
Jul 30 03:47:05.654803 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.654776    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 30 03:47:05.655138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.655120    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 30 03:47:05.661757 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.661736    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"]
Jul 30 03:47:05.706915 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.706819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7d7\" (UniqueName: \"kubernetes.io/projected/6678804d-3999-4dca-a570-dce6d84c1208-kube-api-access-zj7d7\") pod \"payload-pre-processing-687dbb6cf6-vpknv\" (UID: \"6678804d-3999-4dca-a570-dce6d84c1208\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:05.706915 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.706857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6678804d-3999-4dca-a570-dce6d84c1208-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-vpknv\" (UID: \"6678804d-3999-4dca-a570-dce6d84c1208\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:05.808177 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.808138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7d7\" (UniqueName: \"kubernetes.io/projected/6678804d-3999-4dca-a570-dce6d84c1208-kube-api-access-zj7d7\") pod \"payload-pre-processing-687dbb6cf6-vpknv\" (UID: \"6678804d-3999-4dca-a570-dce6d84c1208\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:05.808177 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.808186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6678804d-3999-4dca-a570-dce6d84c1208-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-vpknv\" (UID: \"6678804d-3999-4dca-a570-dce6d84c1208\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:05.808843 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.808824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6678804d-3999-4dca-a570-dce6d84c1208-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-vpknv\" (UID: \"6678804d-3999-4dca-a570-dce6d84c1208\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:05.815451 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.815426    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7d7\" (UniqueName: \"kubernetes.io/projected/6678804d-3999-4dca-a570-dce6d84c1208-kube-api-access-zj7d7\") pod \"payload-pre-processing-687dbb6cf6-vpknv\" (UID: \"6678804d-3999-4dca-a570-dce6d84c1208\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:05.965206 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:05.965116    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:06.100791 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:06.100765    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"]
Jul 30 03:47:06.103251 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:47:06.103214    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6678804d_3999_4dca_a570_dce6d84c1208.slice/crio-b542d0424db6dea5c992ae397ae04777f5268fbc1deb2f5caae715901ea367e0 WatchSource:0}: Error finding container b542d0424db6dea5c992ae397ae04777f5268fbc1deb2f5caae715901ea367e0: Status 404 returned error can't find the container with id b542d0424db6dea5c992ae397ae04777f5268fbc1deb2f5caae715901ea367e0
Jul 30 03:47:06.645596 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:06.645563    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv" event={"ID":"6678804d-3999-4dca-a570-dce6d84c1208","Type":"ContainerStarted","Data":"b542d0424db6dea5c992ae397ae04777f5268fbc1deb2f5caae715901ea367e0"}
Jul 30 03:47:07.652618 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:07.652560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp" event={"ID":"20a7f08d-1113-4c31-837f-0db8f106a7dd","Type":"ContainerStarted","Data":"a4a27a617e9ab0aa39b8656db53f0026b7d2f418fbac5713416afcefe43abfd7"}
Jul 30 03:47:07.653058 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:07.652642    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:07.669468 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:07.669406    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp" podStartSLOduration=0.99651714 podStartE2EDuration="2.66938365s" totalImagesPullingTime="1.67286651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:47:07.667387306 +0000 UTC m=+734.622021988" watchObservedRunningTime="2026-07-30 03:47:07.66938365 +0000 UTC m=+734.624018321"
Jul 30 03:47:08.659885 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:08.659800    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv" event={"ID":"6678804d-3999-4dca-a570-dce6d84c1208","Type":"ContainerStarted","Data":"0f696b0b5e2e36a57c98525a01fe456415caaa5781418e3bcae98bf2af38232d"}
Jul 30 03:47:08.660220 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:08.659895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:08.675596 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:08.675542    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv" podStartSLOduration=1.388923691 podStartE2EDuration="3.675525012s" totalImagesPullingTime="2.286601321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:47:08.673073167 +0000 UTC m=+735.627707836" watchObservedRunningTime="2026-07-30 03:47:08.675525012 +0000 UTC m=+735.630159681"
Jul 30 03:47:13.665113 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:13.665084    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-95d5b8d55-vknqp"
Jul 30 03:47:19.666991 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:19.666963    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-vpknv"
Jul 30 03:47:39.536671 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.536635    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"]
Jul 30 03:47:39.540691 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.540667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.546586 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.546562    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 30 03:47:39.546772 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.546675    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 30 03:47:39.547311 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.547025    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-65bpz\""
Jul 30 03:47:39.547311 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.547054    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 30 03:47:39.558438 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.558409    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"]
Jul 30 03:47:39.616285 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.616227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.616451 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.616308    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8nd\" (UniqueName: \"kubernetes.io/projected/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-kube-api-access-cb8nd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.616451 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.616356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.616451 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.616436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.616564 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.616467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.616564 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.616490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717307 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717488 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717488 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717488 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717488 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717488 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8nd\" (UniqueName: \"kubernetes.io/projected/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-kube-api-access-cb8nd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717793 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717769    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717849 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.717921 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.717901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.719626 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.719607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.719889 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.719869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.725339 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.725317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8nd\" (UniqueName: \"kubernetes.io/projected/fd5bbe32-5507-4d1d-a2c8-b1298ed32085-kube-api-access-cb8nd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-847mw\" (UID: \"fd5bbe32-5507-4d1d-a2c8-b1298ed32085\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.850370 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.850282    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:39.993054 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:39.993020    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"]
Jul 30 03:47:39.996314 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:47:39.996285    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5bbe32_5507_4d1d_a2c8_b1298ed32085.slice/crio-e1c7163a383398dc46d2319eb38101516cfe62341da936d5308acdb2ee00b9dd WatchSource:0}: Error finding container e1c7163a383398dc46d2319eb38101516cfe62341da936d5308acdb2ee00b9dd: Status 404 returned error can't find the container with id e1c7163a383398dc46d2319eb38101516cfe62341da936d5308acdb2ee00b9dd
Jul 30 03:47:40.810599 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:40.810559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw" event={"ID":"fd5bbe32-5507-4d1d-a2c8-b1298ed32085","Type":"ContainerStarted","Data":"e1c7163a383398dc46d2319eb38101516cfe62341da936d5308acdb2ee00b9dd"}
Jul 30 03:47:41.816498 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:41.816462    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw" event={"ID":"fd5bbe32-5507-4d1d-a2c8-b1298ed32085","Type":"ContainerStarted","Data":"ded856a222594cebd9339790a966173266780697c576729ec4090f88512d0ac0"}
Jul 30 03:47:41.841560 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:41.841493    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw" podStartSLOduration=1.641817132 podStartE2EDuration="2.841477389s" totalImagesPullingTime="1.199660257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:47:41.839434858 +0000 UTC m=+768.794069527" watchObservedRunningTime="2026-07-30 03:47:41.841477389 +0000 UTC m=+768.796112057"
Jul 30 03:47:49.850515 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:49.850476    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:49.850960 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:49.850572    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:49.863165 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:49.863138    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:47:50.869961 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:47:50.869931    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-847mw"
Jul 30 03:48:17.028881 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.028794    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"]
Jul 30 03:48:17.035394 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.035367    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.037152 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.037128    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 30 03:48:17.042175 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.042151    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"]
Jul 30 03:48:17.151292 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.151225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.151470 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.151363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.151470 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.151423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f297d50-dd9d-4da9-9e79-86f45e15f023-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.151470 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.151444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5ck2\" (UniqueName: \"kubernetes.io/projected/0f297d50-dd9d-4da9-9e79-86f45e15f023-kube-api-access-n5ck2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.151584 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.151517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.151584 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.151570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.252618 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.252576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.252814 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.252646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.252814 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.252682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f297d50-dd9d-4da9-9e79-86f45e15f023-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.252814 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.252698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5ck2\" (UniqueName: \"kubernetes.io/projected/0f297d50-dd9d-4da9-9e79-86f45e15f023-kube-api-access-n5ck2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.252814 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.252779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.253025 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.252823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.253025 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.252975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.253147 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.253125    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.253198 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.253133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.254973 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.254945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f297d50-dd9d-4da9-9e79-86f45e15f023-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.255364 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.255342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f297d50-dd9d-4da9-9e79-86f45e15f023-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.260633 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.260607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5ck2\" (UniqueName: \"kubernetes.io/projected/0f297d50-dd9d-4da9-9e79-86f45e15f023-kube-api-access-n5ck2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9\" (UID: \"0f297d50-dd9d-4da9-9e79-86f45e15f023\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.347058 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.346964    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:17.495959 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:48:17.495919    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f297d50_dd9d_4da9_9e79_86f45e15f023.slice/crio-af1e74e92b717ad59ab63c58cdcc56de58189c7863ba3a1858a4588914f9a6b7 WatchSource:0}: Error finding container af1e74e92b717ad59ab63c58cdcc56de58189c7863ba3a1858a4588914f9a6b7: Status 404 returned error can't find the container with id af1e74e92b717ad59ab63c58cdcc56de58189c7863ba3a1858a4588914f9a6b7
Jul 30 03:48:17.496099 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.495986    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"]
Jul 30 03:48:17.997089 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.996991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9" event={"ID":"0f297d50-dd9d-4da9-9e79-86f45e15f023","Type":"ContainerStarted","Data":"db3fe17ab66a284aca3b82f2651137aea041218825b10994c5635c7269df248a"}
Jul 30 03:48:17.997089 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:17.997050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9" event={"ID":"0f297d50-dd9d-4da9-9e79-86f45e15f023","Type":"ContainerStarted","Data":"af1e74e92b717ad59ab63c58cdcc56de58189c7863ba3a1858a4588914f9a6b7"}
Jul 30 03:48:18.013025 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:18.012962    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9" podStartSLOduration=0.822116513 podStartE2EDuration="1.012940937s" totalImagesPullingTime="190.824424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:48:18.011870751 +0000 UTC m=+804.966505419" watchObservedRunningTime="2026-07-30 03:48:18.012940937 +0000 UTC m=+804.967575606"
Jul 30 03:48:24.023509 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.023478    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"]
Jul 30 03:48:24.027294 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.027272    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.029262 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.029216    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 30 03:48:24.037645 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.037614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"]
Jul 30 03:48:24.217294 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.217223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.217488 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.217351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.217488 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.217421    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.217488 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.217446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfg42\" (UniqueName: \"kubernetes.io/projected/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-kube-api-access-xfg42\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.217635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.217535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.217635 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.217561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318427 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318427 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfg42\" (UniqueName: \"kubernetes.io/projected/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-kube-api-access-xfg42\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318672 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318672 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318537    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318672 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318672 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318861 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318861 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.318989 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.318959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.320643 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.320622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.320963 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.320943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.325789 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.325766    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfg42\" (UniqueName: \"kubernetes.io/projected/5e93e0ca-19c0-4d3a-b385-18b3b305bc4a-kube-api-access-xfg42\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9vj2\" (UID: \"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.338695 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.338669    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:24.476663 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:24.476631    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"]
Jul 30 03:48:24.478599 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:48:24.478563    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e93e0ca_19c0_4d3a_b385_18b3b305bc4a.slice/crio-9d2e6b8adc0b5a4c3dfd47538bfeeb6639fc57a241e0388da482c8d95b1784d1 WatchSource:0}: Error finding container 9d2e6b8adc0b5a4c3dfd47538bfeeb6639fc57a241e0388da482c8d95b1784d1: Status 404 returned error can't find the container with id 9d2e6b8adc0b5a4c3dfd47538bfeeb6639fc57a241e0388da482c8d95b1784d1
Jul 30 03:48:25.032608 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:25.032515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2" event={"ID":"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a","Type":"ContainerStarted","Data":"e5b8548e640242a2ee0abaaa98c4d51facdddcec4534ae6505975750c950c8c9"}
Jul 30 03:48:25.032608 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:25.032576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2" event={"ID":"5e93e0ca-19c0-4d3a-b385-18b3b305bc4a","Type":"ContainerStarted","Data":"9d2e6b8adc0b5a4c3dfd47538bfeeb6639fc57a241e0388da482c8d95b1784d1"}
Jul 30 03:48:25.049403 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:25.049325    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2" podStartSLOduration=0.872880633 podStartE2EDuration="1.049306682s" totalImagesPullingTime="176.426049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:48:25.047953944 +0000 UTC m=+812.002588636" watchObservedRunningTime="2026-07-30 03:48:25.049306682 +0000 UTC m=+812.003941351"
Jul 30 03:48:27.348017 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:27.347980    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:27.348435 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:27.348064    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:27.360474 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:27.360445    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:28.058786 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:28.058762    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9"
Jul 30 03:48:34.339253 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:34.339187    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:34.339699 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:34.339294    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:34.351596 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:34.351566    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:48:35.093001 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:48:35.092917    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9vj2"
Jul 30 03:50:22.496449 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.496416    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-8b7vq"]
Jul 30 03:50:22.498966 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.496635    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq" podUID="e1f51f45-286b-4591-8758-05f95104d4c2" containerName="manager" containerID="cri-o://66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59" gracePeriod=30
Jul 30 03:50:22.744854 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.744827    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:50:22.804333 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.804227    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1f51f45-286b-4591-8758-05f95104d4c2-webhook-cert\") pod \"e1f51f45-286b-4591-8758-05f95104d4c2\" (UID: \"e1f51f45-286b-4591-8758-05f95104d4c2\") "
Jul 30 03:50:22.804498 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.804375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-484wp\" (UniqueName: \"kubernetes.io/projected/e1f51f45-286b-4591-8758-05f95104d4c2-kube-api-access-484wp\") pod \"e1f51f45-286b-4591-8758-05f95104d4c2\" (UID: \"e1f51f45-286b-4591-8758-05f95104d4c2\") "
Jul 30 03:50:22.806676 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.806643    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f51f45-286b-4591-8758-05f95104d4c2-kube-api-access-484wp" (OuterVolumeSpecName: "kube-api-access-484wp") pod "e1f51f45-286b-4591-8758-05f95104d4c2" (UID: "e1f51f45-286b-4591-8758-05f95104d4c2"). InnerVolumeSpecName "kube-api-access-484wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:50:22.806806 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.806709    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f51f45-286b-4591-8758-05f95104d4c2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e1f51f45-286b-4591-8758-05f95104d4c2" (UID: "e1f51f45-286b-4591-8758-05f95104d4c2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:50:22.905723 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.905686    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-484wp\" (UniqueName: \"kubernetes.io/projected/e1f51f45-286b-4591-8758-05f95104d4c2-kube-api-access-484wp\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:50:22.905723 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:22.905717    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1f51f45-286b-4591-8758-05f95104d4c2-webhook-cert\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:50:23.607002 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.606971    2560 generic.go:356] "Generic (PLEG): container finished" podID="e1f51f45-286b-4591-8758-05f95104d4c2" containerID="66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59" exitCode=0
Jul 30 03:50:23.607400 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.607018    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq"
Jul 30 03:50:23.607400 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.607059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq" event={"ID":"e1f51f45-286b-4591-8758-05f95104d4c2","Type":"ContainerDied","Data":"66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59"}
Jul 30 03:50:23.607400 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.607127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-8b7vq" event={"ID":"e1f51f45-286b-4591-8758-05f95104d4c2","Type":"ContainerDied","Data":"fb1fe4157520880ed0843255faa79259597f85c9049118f26e57f69610942914"}
Jul 30 03:50:23.607400 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.607159    2560 scope.go:117] "RemoveContainer" containerID="66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59"
Jul 30 03:50:23.616253 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.616209    2560 scope.go:117] "RemoveContainer" containerID="66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59"
Jul 30 03:50:23.616510 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:50:23.616490    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59\": container with ID starting with 66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59 not found: ID does not exist" containerID="66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59"
Jul 30 03:50:23.616566 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.616521    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59"} err="failed to get container status \"66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59\": rpc error: code = NotFound desc = could not find container \"66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59\": container with ID starting with 66cc52347dcd34858035a3606511f73102835bba3b3c04ce4d4403ee2d5f6d59 not found: ID does not exist"
Jul 30 03:50:23.624037 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.624013    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-8b7vq"]
Jul 30 03:50:23.629681 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:23.629658    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-8b7vq"]
Jul 30 03:50:25.578010 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:50:25.577967    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f51f45-286b-4591-8758-05f95104d4c2" path="/var/lib/kubelet/pods/e1f51f45-286b-4591-8758-05f95104d4c2/volumes"
Jul 30 03:51:24.388334 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.388257    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-bjls6"]
Jul 30 03:51:24.388834 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.388687    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f51f45-286b-4591-8758-05f95104d4c2" containerName="manager"
Jul 30 03:51:24.388834 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.388702    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f51f45-286b-4591-8758-05f95104d4c2" containerName="manager"
Jul 30 03:51:24.388834 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.388832    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f51f45-286b-4591-8758-05f95104d4c2" containerName="manager"
Jul 30 03:51:24.391995 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.391978    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:24.393992 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.393971    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-674r7\""
Jul 30 03:51:24.393992 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.393987    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 03:51:24.394159 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.394047    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 03:51:24.400388 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.400363    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-bjls6"]
Jul 30 03:51:24.489560 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.489520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjvl\" (UniqueName: \"kubernetes.io/projected/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-kube-api-access-xzjvl\") pod \"maas-controller-5b9bc959cf-bjls6\" (UID: \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\") " pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:24.489751 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.489595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-webhook-cert\") pod \"maas-controller-5b9bc959cf-bjls6\" (UID: \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\") " pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:24.590490 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.590448    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjvl\" (UniqueName: \"kubernetes.io/projected/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-kube-api-access-xzjvl\") pod \"maas-controller-5b9bc959cf-bjls6\" (UID: \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\") " pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:24.590656 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.590539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-webhook-cert\") pod \"maas-controller-5b9bc959cf-bjls6\" (UID: \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\") " pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:24.593117 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.593087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-webhook-cert\") pod \"maas-controller-5b9bc959cf-bjls6\" (UID: \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\") " pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:24.597266 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.597246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjvl\" (UniqueName: \"kubernetes.io/projected/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-kube-api-access-xzjvl\") pod \"maas-controller-5b9bc959cf-bjls6\" (UID: \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\") " pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:24.703763 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.703668    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:24.842360 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.842334    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-bjls6"]
Jul 30 03:51:24.844555 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:51:24.844530    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4b2f42_c5c9_4f7e_b8f9_c523373d72e8.slice/crio-fb9383519d39a70655db00135df32a84a4bec437cdcfccb596d93e1cad7a2047 WatchSource:0}: Error finding container fb9383519d39a70655db00135df32a84a4bec437cdcfccb596d93e1cad7a2047: Status 404 returned error can't find the container with id fb9383519d39a70655db00135df32a84a4bec437cdcfccb596d93e1cad7a2047
Jul 30 03:51:24.910514 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:24.910479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-bjls6" event={"ID":"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8","Type":"ContainerStarted","Data":"fb9383519d39a70655db00135df32a84a4bec437cdcfccb596d93e1cad7a2047"}
Jul 30 03:51:25.917598 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:25.917563    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-bjls6" event={"ID":"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8","Type":"ContainerStarted","Data":"a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b"}
Jul 30 03:51:25.917981 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:25.917719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:25.935054 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:25.934998    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9bc959cf-bjls6" podStartSLOduration=1.413864824 podStartE2EDuration="1.934982397s" totalImagesPullingTime="521.117573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:25.932197047 +0000 UTC m=+992.886831717" watchObservedRunningTime="2026-07-30 03:51:25.934982397 +0000 UTC m=+992.889617068"
Jul 30 03:51:36.925257 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:36.925198    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:41.558070 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:41.558028    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-bjls6"]
Jul 30 03:51:41.558714 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:41.558279    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b9bc959cf-bjls6" podUID="7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8" containerName="manager" containerID="cri-o://a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b" gracePeriod=30
Jul 30 03:51:41.815779 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:41.815714    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:41.939550 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:41.939514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzjvl\" (UniqueName: \"kubernetes.io/projected/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-kube-api-access-xzjvl\") pod \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\" (UID: \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\") "
Jul 30 03:51:41.939744 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:41.939589    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-webhook-cert\") pod \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\" (UID: \"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8\") "
Jul 30 03:51:41.941690 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:41.941658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8" (UID: "7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:51:41.941690 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:41.941658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-kube-api-access-xzjvl" (OuterVolumeSpecName: "kube-api-access-xzjvl") pod "7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8" (UID: "7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8"). InnerVolumeSpecName "kube-api-access-xzjvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:51:42.002436 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.002393    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8" containerID="a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b" exitCode=0
Jul 30 03:51:42.002436 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.002435    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-bjls6"
Jul 30 03:51:42.002692 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.002436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-bjls6" event={"ID":"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8","Type":"ContainerDied","Data":"a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b"}
Jul 30 03:51:42.002692 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.002503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-bjls6" event={"ID":"7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8","Type":"ContainerDied","Data":"fb9383519d39a70655db00135df32a84a4bec437cdcfccb596d93e1cad7a2047"}
Jul 30 03:51:42.002692 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.002533    2560 scope.go:117] "RemoveContainer" containerID="a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b"
Jul 30 03:51:42.012030 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.012014    2560 scope.go:117] "RemoveContainer" containerID="a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b"
Jul 30 03:51:42.012364 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:51:42.012341    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b\": container with ID starting with a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b not found: ID does not exist" containerID="a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b"
Jul 30 03:51:42.012458 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.012369    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b"} err="failed to get container status \"a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b\": rpc error: code = NotFound desc = could not find container \"a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b\": container with ID starting with a4b153d8d9e65705fe27b37b53a7a9d8a2bc9e05360d9fcb3fedb7a156f07b9b not found: ID does not exist"
Jul 30 03:51:42.023582 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.023544    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-bjls6"]
Jul 30 03:51:42.027143 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.027111    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-bjls6"]
Jul 30 03:51:42.040597 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.040567    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-webhook-cert\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:51:42.040597 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.040595    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzjvl\" (UniqueName: \"kubernetes.io/projected/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8-kube-api-access-xzjvl\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:51:42.773914 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.773870    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-b4khk"]
Jul 30 03:51:42.775409 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.775383    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8" containerName="manager"
Jul 30 03:51:42.775540 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.775530    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8" containerName="manager"
Jul 30 03:51:42.775810 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.775797    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8" containerName="manager"
Jul 30 03:51:42.781895 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.781837    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:42.784331 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.784306    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 03:51:42.784477 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.784433    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 03:51:42.784605 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.784586    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-674r7\""
Jul 30 03:51:42.785154 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.785123    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-b4khk"]
Jul 30 03:51:42.847425 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.847384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1303e528-ea13-4f48-9a96-7c52aa717057-webhook-cert\") pod \"maas-controller-5b9bc959cf-b4khk\" (UID: \"1303e528-ea13-4f48-9a96-7c52aa717057\") " pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:42.847600 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.847450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvhkh\" (UniqueName: \"kubernetes.io/projected/1303e528-ea13-4f48-9a96-7c52aa717057-kube-api-access-hvhkh\") pod \"maas-controller-5b9bc959cf-b4khk\" (UID: \"1303e528-ea13-4f48-9a96-7c52aa717057\") " pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:42.947969 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.947931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1303e528-ea13-4f48-9a96-7c52aa717057-webhook-cert\") pod \"maas-controller-5b9bc959cf-b4khk\" (UID: \"1303e528-ea13-4f48-9a96-7c52aa717057\") " pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:42.948129 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.948002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhkh\" (UniqueName: \"kubernetes.io/projected/1303e528-ea13-4f48-9a96-7c52aa717057-kube-api-access-hvhkh\") pod \"maas-controller-5b9bc959cf-b4khk\" (UID: \"1303e528-ea13-4f48-9a96-7c52aa717057\") " pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:42.950541 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.950505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1303e528-ea13-4f48-9a96-7c52aa717057-webhook-cert\") pod \"maas-controller-5b9bc959cf-b4khk\" (UID: \"1303e528-ea13-4f48-9a96-7c52aa717057\") " pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:42.956012 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:42.955981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvhkh\" (UniqueName: \"kubernetes.io/projected/1303e528-ea13-4f48-9a96-7c52aa717057-kube-api-access-hvhkh\") pod \"maas-controller-5b9bc959cf-b4khk\" (UID: \"1303e528-ea13-4f48-9a96-7c52aa717057\") " pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:43.094462 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:43.094362    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:43.233546 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:43.233482    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bc959cf-b4khk"]
Jul 30 03:51:43.235807 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:51:43.235781    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1303e528_ea13_4f48_9a96_7c52aa717057.slice/crio-e404c4cfb8ec45f6e24f8668df4a303b87132c2399be3df45a77d0881334a53f WatchSource:0}: Error finding container e404c4cfb8ec45f6e24f8668df4a303b87132c2399be3df45a77d0881334a53f: Status 404 returned error can't find the container with id e404c4cfb8ec45f6e24f8668df4a303b87132c2399be3df45a77d0881334a53f
Jul 30 03:51:43.237123 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:43.237107    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:51:43.579341 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:43.579306    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8" path="/var/lib/kubelet/pods/7f4b2f42-c5c9-4f7e-b8f9-c523373d72e8/volumes"
Jul 30 03:51:44.015038 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:44.015001    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-b4khk" event={"ID":"1303e528-ea13-4f48-9a96-7c52aa717057","Type":"ContainerStarted","Data":"799d7dcf24408e459dbfc1a42a973356a7ffc79df96e927c6e2cbf51f7d9ba97"}
Jul 30 03:51:44.015502 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:44.015071    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bc959cf-b4khk" event={"ID":"1303e528-ea13-4f48-9a96-7c52aa717057","Type":"ContainerStarted","Data":"e404c4cfb8ec45f6e24f8668df4a303b87132c2399be3df45a77d0881334a53f"}
Jul 30 03:51:44.015502 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:44.015192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:51:44.031071 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:44.031004    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9bc959cf-b4khk" podStartSLOduration=1.650953126 podStartE2EDuration="2.030989109s" totalImagesPullingTime="380.035983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:44.029455366 +0000 UTC m=+1010.984090036" watchObservedRunningTime="2026-07-30 03:51:44.030989109 +0000 UTC m=+1010.985623780"
Jul 30 03:51:55.023174 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:51:55.023141    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9bc959cf-b4khk"
Jul 30 03:58:45.173138 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.173055    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"]
Jul 30 03:58:45.175749 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.173282    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c" podUID="412bb0ea-9f30-4804-a9ef-b805994ae193" containerName="manager" containerID="cri-o://3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a" gracePeriod=10
Jul 30 03:58:45.438164 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.438098    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:58:45.562568 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.562526    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdmc7\" (UniqueName: \"kubernetes.io/projected/412bb0ea-9f30-4804-a9ef-b805994ae193-kube-api-access-pdmc7\") pod \"412bb0ea-9f30-4804-a9ef-b805994ae193\" (UID: \"412bb0ea-9f30-4804-a9ef-b805994ae193\") "
Jul 30 03:58:45.562764 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.562598    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412bb0ea-9f30-4804-a9ef-b805994ae193-extensions-socket-volume\") pod \"412bb0ea-9f30-4804-a9ef-b805994ae193\" (UID: \"412bb0ea-9f30-4804-a9ef-b805994ae193\") "
Jul 30 03:58:45.562968 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.562941    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412bb0ea-9f30-4804-a9ef-b805994ae193-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "412bb0ea-9f30-4804-a9ef-b805994ae193" (UID: "412bb0ea-9f30-4804-a9ef-b805994ae193"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:58:45.564663 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.564640    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412bb0ea-9f30-4804-a9ef-b805994ae193-kube-api-access-pdmc7" (OuterVolumeSpecName: "kube-api-access-pdmc7") pod "412bb0ea-9f30-4804-a9ef-b805994ae193" (UID: "412bb0ea-9f30-4804-a9ef-b805994ae193"). InnerVolumeSpecName "kube-api-access-pdmc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:58:45.663887 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.663852    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdmc7\" (UniqueName: \"kubernetes.io/projected/412bb0ea-9f30-4804-a9ef-b805994ae193-kube-api-access-pdmc7\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:58:45.663887 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:45.663881    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412bb0ea-9f30-4804-a9ef-b805994ae193-extensions-socket-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 03:58:46.092744 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.092705    2560 generic.go:356] "Generic (PLEG): container finished" podID="412bb0ea-9f30-4804-a9ef-b805994ae193" containerID="3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a" exitCode=0
Jul 30 03:58:46.092945 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.092780    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c" event={"ID":"412bb0ea-9f30-4804-a9ef-b805994ae193","Type":"ContainerDied","Data":"3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a"}
Jul 30 03:58:46.092945 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.092796    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"
Jul 30 03:58:46.092945 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.092824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c" event={"ID":"412bb0ea-9f30-4804-a9ef-b805994ae193","Type":"ContainerDied","Data":"6955443792ed18a2f82cd25e19bbf0a1b272e08b6b82d77b2407c64cf96f04b3"}
Jul 30 03:58:46.092945 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.092846    2560 scope.go:117] "RemoveContainer" containerID="3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a"
Jul 30 03:58:46.103421 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.103399    2560 scope.go:117] "RemoveContainer" containerID="3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a"
Jul 30 03:58:46.103749 ip-10-0-137-125 kubenswrapper[2560]: E0730 03:58:46.103725    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a\": container with ID starting with 3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a not found: ID does not exist" containerID="3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a"
Jul 30 03:58:46.103787 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.103760    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a"} err="failed to get container status \"3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a\": rpc error: code = NotFound desc = could not find container \"3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a\": container with ID starting with 3d5f72b6e08a418bbb8176397a9a18e58f18896431cde6eb5ab64d0474ebfe5a not found: ID does not exist"
Jul 30 03:58:46.109053 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.109024    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"]
Jul 30 03:58:46.113023 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:46.112992    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kpt2c"]
Jul 30 03:58:47.578938 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:58:47.578906    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412bb0ea-9f30-4804-a9ef-b805994ae193" path="/var/lib/kubelet/pods/412bb0ea-9f30-4804-a9ef-b805994ae193/volumes"
Jul 30 03:59:45.815665 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.815630    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"]
Jul 30 03:59:45.816135 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.816019    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412bb0ea-9f30-4804-a9ef-b805994ae193" containerName="manager"
Jul 30 03:59:45.816135 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.816031    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="412bb0ea-9f30-4804-a9ef-b805994ae193" containerName="manager"
Jul 30 03:59:45.816135 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.816098    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="412bb0ea-9f30-4804-a9ef-b805994ae193" containerName="manager"
Jul 30 03:59:45.819458 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.819437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:45.821581 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.821563    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6v4xr\""
Jul 30 03:59:45.831260 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.831206    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"]
Jul 30 03:59:45.914300 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.914262    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh586\" (UniqueName: \"kubernetes.io/projected/755e8acc-c6ee-4577-b882-8ae74307929e-kube-api-access-rh586\") pod \"kuadrant-operator-controller-manager-86878c769f-5kj96\" (UID: \"755e8acc-c6ee-4577-b882-8ae74307929e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:45.914496 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:45.914388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/755e8acc-c6ee-4577-b882-8ae74307929e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5kj96\" (UID: \"755e8acc-c6ee-4577-b882-8ae74307929e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:46.015739 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.015707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/755e8acc-c6ee-4577-b882-8ae74307929e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5kj96\" (UID: \"755e8acc-c6ee-4577-b882-8ae74307929e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:46.015927 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.015771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh586\" (UniqueName: \"kubernetes.io/projected/755e8acc-c6ee-4577-b882-8ae74307929e-kube-api-access-rh586\") pod \"kuadrant-operator-controller-manager-86878c769f-5kj96\" (UID: \"755e8acc-c6ee-4577-b882-8ae74307929e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:46.016137 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.016112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/755e8acc-c6ee-4577-b882-8ae74307929e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5kj96\" (UID: \"755e8acc-c6ee-4577-b882-8ae74307929e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:46.024256 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.024198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh586\" (UniqueName: \"kubernetes.io/projected/755e8acc-c6ee-4577-b882-8ae74307929e-kube-api-access-rh586\") pod \"kuadrant-operator-controller-manager-86878c769f-5kj96\" (UID: \"755e8acc-c6ee-4577-b882-8ae74307929e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:46.131175 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.131080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:46.270935 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.270905    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"]
Jul 30 03:59:46.273808 ip-10-0-137-125 kubenswrapper[2560]: W0730 03:59:46.273772    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755e8acc_c6ee_4577_b882_8ae74307929e.slice/crio-a45894c6c5a50a6f943c0deaa9a0346d7c935abee51620333e9cff638cfe6ea3 WatchSource:0}: Error finding container a45894c6c5a50a6f943c0deaa9a0346d7c935abee51620333e9cff638cfe6ea3: Status 404 returned error can't find the container with id a45894c6c5a50a6f943c0deaa9a0346d7c935abee51620333e9cff638cfe6ea3
Jul 30 03:59:46.382702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.382598    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96" event={"ID":"755e8acc-c6ee-4577-b882-8ae74307929e","Type":"ContainerStarted","Data":"677c241d308d87cb04b8a39b6042a5bacedfa50b7bbac8f2b95110b1ee73ade1"}
Jul 30 03:59:46.382702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.382664    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96" event={"ID":"755e8acc-c6ee-4577-b882-8ae74307929e","Type":"ContainerStarted","Data":"a45894c6c5a50a6f943c0deaa9a0346d7c935abee51620333e9cff638cfe6ea3"}
Jul 30 03:59:46.382702 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.382693    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 03:59:46.399761 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:46.399692    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96" podStartSLOduration=1.399671044 podStartE2EDuration="1.399671044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:59:46.398708469 +0000 UTC m=+1493.353343166" watchObservedRunningTime="2026-07-30 03:59:46.399671044 +0000 UTC m=+1493.354305716"
Jul 30 03:59:57.389965 ip-10-0-137-125 kubenswrapper[2560]: I0730 03:59:57.389934    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 04:00:00.138436 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:00.138399    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9"]
Jul 30 04:00:00.142472 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:00.142451    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9"
Jul 30 04:00:00.144304 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:00.144277    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:00:00.148944 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:00.148918    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9"]
Jul 30 04:00:00.153441 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:00.153412    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9"
Jul 30 04:00:00.288954 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:00.288924    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9"]
Jul 30 04:00:00.292711 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:00:00.292678    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2e0ebe_2cfd_4657_9f2c_8ef6e2f996a6.slice/crio-8a9df656da6088fae268bc3695feae02394ddf4fcd0d44f50757df5a4a4ca27c WatchSource:0}: Error finding container 8a9df656da6088fae268bc3695feae02394ddf4fcd0d44f50757df5a4a4ca27c: Status 404 returned error can't find the container with id 8a9df656da6088fae268bc3695feae02394ddf4fcd0d44f50757df5a4a4ca27c
Jul 30 04:00:00.294695 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:00.294680    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:00:00.451108 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:00.451011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9" event={"ID":"be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6","Type":"ContainerStarted","Data":"8a9df656da6088fae268bc3695feae02394ddf4fcd0d44f50757df5a4a4ca27c"}
Jul 30 04:00:03.470771 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:03.470718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9" event={"ID":"be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6","Type":"ContainerStarted","Data":"77a0ce13f5eb1a9dccbe50bd426b051da13e7d020367d11a2b7dec190d0931b7"}
Jul 30 04:00:03.483750 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:03.483458    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9" podStartSLOduration=1.159045352 podStartE2EDuration="3.483437266s" totalImagesPullingTime="2.324391914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:00:03.483008169 +0000 UTC m=+1510.437642841" watchObservedRunningTime="2026-07-30 04:00:03.483437266 +0000 UTC m=+1510.438071941"
Jul 30 04:00:04.477346 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:04.477303    2560 generic.go:356] "Generic (PLEG): container finished" podID="be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6" containerID="77a0ce13f5eb1a9dccbe50bd426b051da13e7d020367d11a2b7dec190d0931b7" exitCode=0
Jul 30 04:00:04.477788 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:04.477384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9" event={"ID":"be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6","Type":"ContainerDied","Data":"77a0ce13f5eb1a9dccbe50bd426b051da13e7d020367d11a2b7dec190d0931b7"}
Jul 30 04:00:05.615370 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:05.615342    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9"
Jul 30 04:00:06.488173 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:06.488141    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9"
Jul 30 04:00:06.488173 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:06.488155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756400-5btw9" event={"ID":"be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6","Type":"ContainerDied","Data":"8a9df656da6088fae268bc3695feae02394ddf4fcd0d44f50757df5a4a4ca27c"}
Jul 30 04:00:06.488389 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:00:06.488200    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a9df656da6088fae268bc3695feae02394ddf4fcd0d44f50757df5a4a4ca27c"
Jul 30 04:08:01.169920 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.169837    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"]
Jul 30 04:08:01.170565 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.170541    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6" containerName="cleanup"
Jul 30 04:08:01.170653 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.170568    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6" containerName="cleanup"
Jul 30 04:08:01.170710 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.170667    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6" containerName="cleanup"
Jul 30 04:08:01.173864 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.173843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.175602 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.175576    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0a082c3b-serving-cert\""
Jul 30 04:08:01.185316 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.184960    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"]
Jul 30 04:08:01.334150 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.334109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cafc163-2466-4f27-a19c-8a6291ec82e8-openshift-service-ca\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.334377 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.334344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crk6f\" (UniqueName: \"kubernetes.io/projected/3cafc163-2466-4f27-a19c-8a6291ec82e8-kube-api-access-crk6f\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.334452 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.334383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cafc163-2466-4f27-a19c-8a6291ec82e8-maas-api-tls\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.435663 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.435574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crk6f\" (UniqueName: \"kubernetes.io/projected/3cafc163-2466-4f27-a19c-8a6291ec82e8-kube-api-access-crk6f\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.435663 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.435617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cafc163-2466-4f27-a19c-8a6291ec82e8-maas-api-tls\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.435663 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.435643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cafc163-2466-4f27-a19c-8a6291ec82e8-openshift-service-ca\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.436402 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.436384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cafc163-2466-4f27-a19c-8a6291ec82e8-openshift-service-ca\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.438124 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.438105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cafc163-2466-4f27-a19c-8a6291ec82e8-maas-api-tls\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.442513 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.442494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crk6f\" (UniqueName: \"kubernetes.io/projected/3cafc163-2466-4f27-a19c-8a6291ec82e8-kube-api-access-crk6f\") pod \"maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.488193 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.488155    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:01.627757 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.627733    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"]
Jul 30 04:08:01.630286 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:08:01.630226    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cafc163_2466_4f27_a19c_8a6291ec82e8.slice/crio-7f5b55c25967e83c7d9f9187e57a854ea994e8b71ff59800c49a9b6549c1ac63 WatchSource:0}: Error finding container 7f5b55c25967e83c7d9f9187e57a854ea994e8b71ff59800c49a9b6549c1ac63: Status 404 returned error can't find the container with id 7f5b55c25967e83c7d9f9187e57a854ea994e8b71ff59800c49a9b6549c1ac63
Jul 30 04:08:01.631643 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.631623    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:08:01.767705 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.767568    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"]
Jul 30 04:08:01.773604 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.773575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:01.776171 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.776145    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0a082c3b\""
Jul 30 04:08:01.776415 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.776390    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0a082c3b-dockercfg-prrmj\""
Jul 30 04:08:01.783885 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.783827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"]
Jul 30 04:08:01.838899 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.838864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702d00e-d7fc-44a7-a070-a468025ce15b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz\" (UID: \"f702d00e-d7fc-44a7-a070-a468025ce15b\") " pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:01.839081 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.838903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7c4r\" (UniqueName: \"kubernetes.io/projected/f702d00e-d7fc-44a7-a070-a468025ce15b-kube-api-access-q7c4r\") pod \"payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz\" (UID: \"f702d00e-d7fc-44a7-a070-a468025ce15b\") " pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:01.872910 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.872856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j" event={"ID":"3cafc163-2466-4f27-a19c-8a6291ec82e8","Type":"ContainerStarted","Data":"7f5b55c25967e83c7d9f9187e57a854ea994e8b71ff59800c49a9b6549c1ac63"}
Jul 30 04:08:01.940212 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.940173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702d00e-d7fc-44a7-a070-a468025ce15b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz\" (UID: \"f702d00e-d7fc-44a7-a070-a468025ce15b\") " pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:01.940212 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.940211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7c4r\" (UniqueName: \"kubernetes.io/projected/f702d00e-d7fc-44a7-a070-a468025ce15b-kube-api-access-q7c4r\") pod \"payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz\" (UID: \"f702d00e-d7fc-44a7-a070-a468025ce15b\") " pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:01.940930 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.940904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702d00e-d7fc-44a7-a070-a468025ce15b-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz\" (UID: \"f702d00e-d7fc-44a7-a070-a468025ce15b\") " pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:01.952355 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:01.952320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7c4r\" (UniqueName: \"kubernetes.io/projected/f702d00e-d7fc-44a7-a070-a468025ce15b-kube-api-access-q7c4r\") pod \"payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz\" (UID: \"f702d00e-d7fc-44a7-a070-a468025ce15b\") " pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:02.087942 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.087857    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:02.224363 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.224338    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"]
Jul 30 04:08:02.226799 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:08:02.226764    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf702d00e_d7fc_44a7_a070_a468025ce15b.slice/crio-cafbaacf9110b457069643cfb1278ac0c58f8f6b454ff148f761c3461e5b974f WatchSource:0}: Error finding container cafbaacf9110b457069643cfb1278ac0c58f8f6b454ff148f761c3461e5b974f: Status 404 returned error can't find the container with id cafbaacf9110b457069643cfb1278ac0c58f8f6b454ff148f761c3461e5b974f
Jul 30 04:08:02.881042 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.881002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j" event={"ID":"3cafc163-2466-4f27-a19c-8a6291ec82e8","Type":"ContainerStarted","Data":"5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb"}
Jul 30 04:08:02.881289 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.881071    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:02.882820 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.882788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz" event={"ID":"f702d00e-d7fc-44a7-a070-a468025ce15b","Type":"ContainerStarted","Data":"be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1"}
Jul 30 04:08:02.882956 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.882844    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz" event={"ID":"f702d00e-d7fc-44a7-a070-a468025ce15b","Type":"ContainerStarted","Data":"cafbaacf9110b457069643cfb1278ac0c58f8f6b454ff148f761c3461e5b974f"}
Jul 30 04:08:02.882956 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.882885    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:02.897279 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.897206    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j" podStartSLOduration=1.202186298 podStartE2EDuration="1.89718229s" totalImagesPullingTime="694.995992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:08:02.895311073 +0000 UTC m=+1989.849945745" watchObservedRunningTime="2026-07-30 04:08:02.89718229 +0000 UTC m=+1989.851816962"
Jul 30 04:08:02.910014 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:02.909964    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz" podStartSLOduration=1.90994869 podStartE2EDuration="1.90994869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:08:02.907715759 +0000 UTC m=+1989.862350429" watchObservedRunningTime="2026-07-30 04:08:02.90994869 +0000 UTC m=+1989.864583361"
Jul 30 04:08:08.896741 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:08.896707    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:13.892026 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:13.891991    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:25.540803 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:25.540765    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"]
Jul 30 04:08:25.545813 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:25.545793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"
Jul 30 04:08:25.547531 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:25.547509    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:08:25.551006 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:25.550977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"]
Jul 30 04:08:25.561007 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:25.560981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4157d291-ab2a-4bba-88c2-1273c0a494e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5\" (UID: \"4157d291-ab2a-4bba-88c2-1273c0a494e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"
Jul 30 04:08:25.662318 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:25.662276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4157d291-ab2a-4bba-88c2-1273c0a494e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5\" (UID: \"4157d291-ab2a-4bba-88c2-1273c0a494e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"
Jul 30 04:08:25.662895 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:25.662873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4157d291-ab2a-4bba-88c2-1273c0a494e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5\" (UID: \"4157d291-ab2a-4bba-88c2-1273c0a494e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"
Jul 30 04:08:25.858994 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:25.858903    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"
Jul 30 04:08:26.007552 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:26.007524    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"]
Jul 30 04:08:26.011019 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:08:26.010983    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4157d291_ab2a_4bba_88c2_1273c0a494e3.slice/crio-25a87f3e9b0a69bdf7dfd8b28d04ae8ae1f61209b09aa86e5cc0dea31f55eb57 WatchSource:0}: Error finding container 25a87f3e9b0a69bdf7dfd8b28d04ae8ae1f61209b09aa86e5cc0dea31f55eb57: Status 404 returned error can't find the container with id 25a87f3e9b0a69bdf7dfd8b28d04ae8ae1f61209b09aa86e5cc0dea31f55eb57
Jul 30 04:08:27.008995 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:27.008907    2560 generic.go:356] "Generic (PLEG): container finished" podID="4157d291-ab2a-4bba-88c2-1273c0a494e3" containerID="81df84dd66825fd272a8e4986857a7439a644ce638b47781f529bd559a6b3982" exitCode=0
Jul 30 04:08:27.009384 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:27.008995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5" event={"ID":"4157d291-ab2a-4bba-88c2-1273c0a494e3","Type":"ContainerDied","Data":"81df84dd66825fd272a8e4986857a7439a644ce638b47781f529bd559a6b3982"}
Jul 30 04:08:27.009384 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:27.009055    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5" event={"ID":"4157d291-ab2a-4bba-88c2-1273c0a494e3","Type":"ContainerStarted","Data":"25a87f3e9b0a69bdf7dfd8b28d04ae8ae1f61209b09aa86e5cc0dea31f55eb57"}
Jul 30 04:08:28.148792 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:28.148765    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"
Jul 30 04:08:28.189229 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:28.189193    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4157d291-ab2a-4bba-88c2-1273c0a494e3-maas-api-service-ca\") pod \"4157d291-ab2a-4bba-88c2-1273c0a494e3\" (UID: \"4157d291-ab2a-4bba-88c2-1273c0a494e3\") "
Jul 30 04:08:28.189572 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:28.189542    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4157d291-ab2a-4bba-88c2-1273c0a494e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4157d291-ab2a-4bba-88c2-1273c0a494e3" (UID: "4157d291-ab2a-4bba-88c2-1273c0a494e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:08:28.290456 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:28.290365    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4157d291-ab2a-4bba-88c2-1273c0a494e3-maas-api-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:08:29.019856 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:29.019824    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"
Jul 30 04:08:29.019856 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:29.019842    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5" event={"ID":"4157d291-ab2a-4bba-88c2-1273c0a494e3","Type":"ContainerDied","Data":"25a87f3e9b0a69bdf7dfd8b28d04ae8ae1f61209b09aa86e5cc0dea31f55eb57"}
Jul 30 04:08:29.020075 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:29.019888    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a87f3e9b0a69bdf7dfd8b28d04ae8ae1f61209b09aa86e5cc0dea31f55eb57"
Jul 30 04:08:31.533854 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.533808    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"]
Jul 30 04:08:31.534383 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.534131    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j" podUID="3cafc163-2466-4f27-a19c-8a6291ec82e8" containerName="maas-api" containerID="cri-o://5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb" gracePeriod=30
Jul 30 04:08:31.587581 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.587550    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"]
Jul 30 04:08:31.587866 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.587824    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz" podUID="f702d00e-d7fc-44a7-a070-a468025ce15b" containerName="payload-processing" containerID="cri-o://be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1" gracePeriod=30
Jul 30 04:08:31.756845 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.756812    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"]
Jul 30 04:08:31.762329 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.760362    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a082c3b-bqtz5"]
Jul 30 04:08:31.839552 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.839523    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:31.871529 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.871499    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:31.926466 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.926426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cafc163-2466-4f27-a19c-8a6291ec82e8-openshift-service-ca\") pod \"3cafc163-2466-4f27-a19c-8a6291ec82e8\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") "
Jul 30 04:08:31.926667 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.926474    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crk6f\" (UniqueName: \"kubernetes.io/projected/3cafc163-2466-4f27-a19c-8a6291ec82e8-kube-api-access-crk6f\") pod \"3cafc163-2466-4f27-a19c-8a6291ec82e8\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") "
Jul 30 04:08:31.926667 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.926545    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cafc163-2466-4f27-a19c-8a6291ec82e8-maas-api-tls\") pod \"3cafc163-2466-4f27-a19c-8a6291ec82e8\" (UID: \"3cafc163-2466-4f27-a19c-8a6291ec82e8\") "
Jul 30 04:08:31.926667 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.926577    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7c4r\" (UniqueName: \"kubernetes.io/projected/f702d00e-d7fc-44a7-a070-a468025ce15b-kube-api-access-q7c4r\") pod \"f702d00e-d7fc-44a7-a070-a468025ce15b\" (UID: \"f702d00e-d7fc-44a7-a070-a468025ce15b\") "
Jul 30 04:08:31.926667 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.926628    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702d00e-d7fc-44a7-a070-a468025ce15b-plugins-config-volume\") pod \"f702d00e-d7fc-44a7-a070-a468025ce15b\" (UID: \"f702d00e-d7fc-44a7-a070-a468025ce15b\") "
Jul 30 04:08:31.926880 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.926776    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cafc163-2466-4f27-a19c-8a6291ec82e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cafc163-2466-4f27-a19c-8a6291ec82e8" (UID: "3cafc163-2466-4f27-a19c-8a6291ec82e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:08:31.926941 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.926930    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cafc163-2466-4f27-a19c-8a6291ec82e8-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:08:31.927086 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.927058    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f702d00e-d7fc-44a7-a070-a468025ce15b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f702d00e-d7fc-44a7-a070-a468025ce15b" (UID: "f702d00e-d7fc-44a7-a070-a468025ce15b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:08:31.928803 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.928773    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cafc163-2466-4f27-a19c-8a6291ec82e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cafc163-2466-4f27-a19c-8a6291ec82e8" (UID: "3cafc163-2466-4f27-a19c-8a6291ec82e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:08:31.928913 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.928823    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cafc163-2466-4f27-a19c-8a6291ec82e8-kube-api-access-crk6f" (OuterVolumeSpecName: "kube-api-access-crk6f") pod "3cafc163-2466-4f27-a19c-8a6291ec82e8" (UID: "3cafc163-2466-4f27-a19c-8a6291ec82e8"). InnerVolumeSpecName "kube-api-access-crk6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:08:31.928913 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:31.928848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f702d00e-d7fc-44a7-a070-a468025ce15b-kube-api-access-q7c4r" (OuterVolumeSpecName: "kube-api-access-q7c4r") pod "f702d00e-d7fc-44a7-a070-a468025ce15b" (UID: "f702d00e-d7fc-44a7-a070-a468025ce15b"). InnerVolumeSpecName "kube-api-access-q7c4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:08:32.027641 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.027599    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crk6f\" (UniqueName: \"kubernetes.io/projected/3cafc163-2466-4f27-a19c-8a6291ec82e8-kube-api-access-crk6f\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:08:32.027641 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.027631    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cafc163-2466-4f27-a19c-8a6291ec82e8-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:08:32.027641 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.027643    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7c4r\" (UniqueName: \"kubernetes.io/projected/f702d00e-d7fc-44a7-a070-a468025ce15b-kube-api-access-q7c4r\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:08:32.027641 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.027652    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f702d00e-d7fc-44a7-a070-a468025ce15b-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:08:32.035012 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.034977    2560 generic.go:356] "Generic (PLEG): container finished" podID="3cafc163-2466-4f27-a19c-8a6291ec82e8" containerID="5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb" exitCode=0
Jul 30 04:08:32.035195 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.035027    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"
Jul 30 04:08:32.035195 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.035075    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j" event={"ID":"3cafc163-2466-4f27-a19c-8a6291ec82e8","Type":"ContainerDied","Data":"5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb"}
Jul 30 04:08:32.035195 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.035138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j" event={"ID":"3cafc163-2466-4f27-a19c-8a6291ec82e8","Type":"ContainerDied","Data":"7f5b55c25967e83c7d9f9187e57a854ea994e8b71ff59800c49a9b6549c1ac63"}
Jul 30 04:08:32.035195 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.035172    2560 scope.go:117] "RemoveContainer" containerID="5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb"
Jul 30 04:08:32.036608 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.036581    2560 generic.go:356] "Generic (PLEG): container finished" podID="f702d00e-d7fc-44a7-a070-a468025ce15b" containerID="be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1" exitCode=0
Jul 30 04:08:32.036694 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.036633    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"
Jul 30 04:08:32.036741 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.036644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz" event={"ID":"f702d00e-d7fc-44a7-a070-a468025ce15b","Type":"ContainerDied","Data":"be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1"}
Jul 30 04:08:32.036799 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.036779    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz" event={"ID":"f702d00e-d7fc-44a7-a070-a468025ce15b","Type":"ContainerDied","Data":"cafbaacf9110b457069643cfb1278ac0c58f8f6b454ff148f761c3461e5b974f"}
Jul 30 04:08:32.044979 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.044957    2560 scope.go:117] "RemoveContainer" containerID="5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb"
Jul 30 04:08:32.045341 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:08:32.045320    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb\": container with ID starting with 5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb not found: ID does not exist" containerID="5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb"
Jul 30 04:08:32.045420 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.045353    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb"} err="failed to get container status \"5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb\": rpc error: code = NotFound desc = could not find container \"5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb\": container with ID starting with 5df964b6b83ffb519e29fa4e22dfd1590a9b09ac3ee9ca518d5c3877ab4111fb not found: ID does not exist"
Jul 30 04:08:32.045420 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.045373    2560 scope.go:117] "RemoveContainer" containerID="be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1"
Jul 30 04:08:32.054011 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.053985    2560 scope.go:117] "RemoveContainer" containerID="be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1"
Jul 30 04:08:32.054342 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:08:32.054323    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1\": container with ID starting with be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1 not found: ID does not exist" containerID="be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1"
Jul 30 04:08:32.054403 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.054350    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1"} err="failed to get container status \"be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1\": rpc error: code = NotFound desc = could not find container \"be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1\": container with ID starting with be36a928800c14bb7b75b65cda808109d088b9389aa9bc1b456d1bd8beb45ce1 not found: ID does not exist"
Jul 30 04:08:32.058814 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.058784    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"]
Jul 30 04:08:32.063980 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.063955    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a082c3b-677d796cd-tw5hz"]
Jul 30 04:08:32.073779 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.073739    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"]
Jul 30 04:08:32.078971 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:32.078941    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a082c3b-6c8768c889-dnx8j"]
Jul 30 04:08:33.578821 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:33.578788    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cafc163-2466-4f27-a19c-8a6291ec82e8" path="/var/lib/kubelet/pods/3cafc163-2466-4f27-a19c-8a6291ec82e8/volumes"
Jul 30 04:08:33.579209 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:33.579128    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4157d291-ab2a-4bba-88c2-1273c0a494e3" path="/var/lib/kubelet/pods/4157d291-ab2a-4bba-88c2-1273c0a494e3/volumes"
Jul 30 04:08:33.579464 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:33.579450    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f702d00e-d7fc-44a7-a070-a468025ce15b" path="/var/lib/kubelet/pods/f702d00e-d7fc-44a7-a070-a468025ce15b/volumes"
Jul 30 04:08:35.374120 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374068    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"]
Jul 30 04:08:35.374785 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374761    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4157d291-ab2a-4bba-88c2-1273c0a494e3" containerName="revoke-keys"
Jul 30 04:08:35.374879 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374789    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4157d291-ab2a-4bba-88c2-1273c0a494e3" containerName="revoke-keys"
Jul 30 04:08:35.374879 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374835    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f702d00e-d7fc-44a7-a070-a468025ce15b" containerName="payload-processing"
Jul 30 04:08:35.374879 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374844    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f702d00e-d7fc-44a7-a070-a468025ce15b" containerName="payload-processing"
Jul 30 04:08:35.374879 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374862    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cafc163-2466-4f27-a19c-8a6291ec82e8" containerName="maas-api"
Jul 30 04:08:35.374879 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374871    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cafc163-2466-4f27-a19c-8a6291ec82e8" containerName="maas-api"
Jul 30 04:08:35.375141 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374956    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cafc163-2466-4f27-a19c-8a6291ec82e8" containerName="maas-api"
Jul 30 04:08:35.375141 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374971    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4157d291-ab2a-4bba-88c2-1273c0a494e3" containerName="revoke-keys"
Jul 30 04:08:35.375141 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.374981    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f702d00e-d7fc-44a7-a070-a468025ce15b" containerName="payload-processing"
Jul 30 04:08:35.379952 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.379920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:35.381788 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.381764    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e1a4694c-serving-cert\""
Jul 30 04:08:35.386533 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.386505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"]
Jul 30 04:08:35.465052 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.465002    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b6a27-7ae3-4dc0-bc64-07f49fa46983-openshift-service-ca\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:35.465220 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.465086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:35.465320 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.465211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxnp5\" (UniqueName: \"kubernetes.io/projected/699b6a27-7ae3-4dc0-bc64-07f49fa46983-kube-api-access-dxnp5\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:35.566662 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.566613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b6a27-7ae3-4dc0-bc64-07f49fa46983-openshift-service-ca\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:35.566842 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.566678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:35.566842 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.566738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxnp5\" (UniqueName: \"kubernetes.io/projected/699b6a27-7ae3-4dc0-bc64-07f49fa46983-kube-api-access-dxnp5\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:35.566931 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:08:35.566863    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-serving-cert: secret "maas-api-e2e-migrate-e1a4694c-serving-cert" not found
Jul 30 04:08:35.566964 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:08:35.566943    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls podName:699b6a27-7ae3-4dc0-bc64-07f49fa46983 nodeName:}" failed. No retries permitted until 2026-07-30 04:08:36.066921439 +0000 UTC m=+2023.021556094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls") pod "maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw" (UID: "699b6a27-7ae3-4dc0-bc64-07f49fa46983") : secret "maas-api-e2e-migrate-e1a4694c-serving-cert" not found
Jul 30 04:08:35.567300 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.567267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b6a27-7ae3-4dc0-bc64-07f49fa46983-openshift-service-ca\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:35.576440 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:35.576406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxnp5\" (UniqueName: \"kubernetes.io/projected/699b6a27-7ae3-4dc0-bc64-07f49fa46983-kube-api-access-dxnp5\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:36.009970 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.009926    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"]
Jul 30 04:08:36.013827 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.013808    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:36.015552 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.015532    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e1a4694c-dockercfg-9rd54\""
Jul 30 04:08:36.015802 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.015785    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e1a4694c\""
Jul 30 04:08:36.026762 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.026733    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"]
Jul 30 04:08:36.042103 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.042064    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"]
Jul 30 04:08:36.046313 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.046294    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:36.052657 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.052555    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"]
Jul 30 04:08:36.074389 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.074351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:36.074566 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.074486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn\" (UID: \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:36.074566 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:08:36.074516    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-serving-cert: secret "maas-api-e2e-migrate-e1a4694c-serving-cert" not found
Jul 30 04:08:36.074694 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.074522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vwq\" (UniqueName: \"kubernetes.io/projected/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-kube-api-access-48vwq\") pod \"payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn\" (UID: \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:36.074694 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:08:36.074590    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls podName:699b6a27-7ae3-4dc0-bc64-07f49fa46983 nodeName:}" failed. No retries permitted until 2026-07-30 04:08:37.074567858 +0000 UTC m=+2024.029202511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls") pod "maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw" (UID: "699b6a27-7ae3-4dc0-bc64-07f49fa46983") : secret "maas-api-e2e-migrate-e1a4694c-serving-cert" not found
Jul 30 04:08:36.175584 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.175543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c85f24-6019-4f24-904e-6bceff237f82-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck\" (UID: \"48c85f24-6019-4f24-904e-6bceff237f82\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:36.175584 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.175582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjgb\" (UniqueName: \"kubernetes.io/projected/48c85f24-6019-4f24-904e-6bceff237f82-kube-api-access-7rjgb\") pod \"payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck\" (UID: \"48c85f24-6019-4f24-904e-6bceff237f82\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:36.175858 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.175692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn\" (UID: \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:36.175858 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.175737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vwq\" (UniqueName: \"kubernetes.io/projected/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-kube-api-access-48vwq\") pod \"payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn\" (UID: \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:36.176550 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.176525    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn\" (UID: \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:36.182840 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.182813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vwq\" (UniqueName: \"kubernetes.io/projected/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-kube-api-access-48vwq\") pod \"payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn\" (UID: \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:36.277287 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.277160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c85f24-6019-4f24-904e-6bceff237f82-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck\" (UID: \"48c85f24-6019-4f24-904e-6bceff237f82\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:36.277287 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.277201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjgb\" (UniqueName: \"kubernetes.io/projected/48c85f24-6019-4f24-904e-6bceff237f82-kube-api-access-7rjgb\") pod \"payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck\" (UID: \"48c85f24-6019-4f24-904e-6bceff237f82\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:36.278309 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.278281    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c85f24-6019-4f24-904e-6bceff237f82-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck\" (UID: \"48c85f24-6019-4f24-904e-6bceff237f82\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:36.285174 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.285139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjgb\" (UniqueName: \"kubernetes.io/projected/48c85f24-6019-4f24-904e-6bceff237f82-kube-api-access-7rjgb\") pod \"payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck\" (UID: \"48c85f24-6019-4f24-904e-6bceff237f82\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:36.325124 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.325085    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:36.360384 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.360352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:36.486358 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.486316    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"]
Jul 30 04:08:36.489855 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:08:36.489816    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aacd380_f4b2_45ca_b7f1_f7e7890310f5.slice/crio-626153bcb472acbef2e6c626e88443a16a83691f7cbe75490b9f7ff091a4e704 WatchSource:0}: Error finding container 626153bcb472acbef2e6c626e88443a16a83691f7cbe75490b9f7ff091a4e704: Status 404 returned error can't find the container with id 626153bcb472acbef2e6c626e88443a16a83691f7cbe75490b9f7ff091a4e704
Jul 30 04:08:36.533699 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:36.533672    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"]
Jul 30 04:08:36.536440 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:08:36.536402    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c85f24_6019_4f24_904e_6bceff237f82.slice/crio-eab67251d40b3bef59c1b7ad477bca60bdaa5210e8d8b2567eb1b40d65478794 WatchSource:0}: Error finding container eab67251d40b3bef59c1b7ad477bca60bdaa5210e8d8b2567eb1b40d65478794: Status 404 returned error can't find the container with id eab67251d40b3bef59c1b7ad477bca60bdaa5210e8d8b2567eb1b40d65478794
Jul 30 04:08:37.069772 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.069732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn" event={"ID":"3aacd380-f4b2-45ca-b7f1-f7e7890310f5","Type":"ContainerStarted","Data":"ab1c0b4e88c39354ee3c6257c6f933e0b39b5b383a9f77e1fd9c2040e7f2d4ae"}
Jul 30 04:08:37.070400 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.070373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn" event={"ID":"3aacd380-f4b2-45ca-b7f1-f7e7890310f5","Type":"ContainerStarted","Data":"626153bcb472acbef2e6c626e88443a16a83691f7cbe75490b9f7ff091a4e704"}
Jul 30 04:08:37.070591 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.070580    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:37.072038 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.072010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck" event={"ID":"48c85f24-6019-4f24-904e-6bceff237f82","Type":"ContainerStarted","Data":"10a762b45c336a9ec91a5ee81fedf1705618e85f48ea1b5d06be5a15d75724e3"}
Jul 30 04:08:37.072164 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.072057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck" event={"ID":"48c85f24-6019-4f24-904e-6bceff237f82","Type":"ContainerStarted","Data":"eab67251d40b3bef59c1b7ad477bca60bdaa5210e8d8b2567eb1b40d65478794"}
Jul 30 04:08:37.072614 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.072595    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:08:37.084281 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.084220    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:37.086809 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.086751    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn" podStartSLOduration=2.086730402 podStartE2EDuration="2.086730402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:08:37.084060121 +0000 UTC m=+2024.038694782" watchObservedRunningTime="2026-07-30 04:08:37.086730402 +0000 UTC m=+2024.041365078"
Jul 30 04:08:37.087519 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.087491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls\") pod \"maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:37.101207 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.101115    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck" podStartSLOduration=1.101094703 podStartE2EDuration="1.101094703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:08:37.098428476 +0000 UTC m=+2024.053063150" watchObservedRunningTime="2026-07-30 04:08:37.101094703 +0000 UTC m=+2024.055729377"
Jul 30 04:08:37.193935 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.193900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:37.339104 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:37.338899    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"]
Jul 30 04:08:37.341799 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:08:37.341767    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699b6a27_7ae3_4dc0_bc64_07f49fa46983.slice/crio-3fbd5e047f1078ca3dbb5b5843d8497b2108b5bd97af187f2a03d7aebbd58f03 WatchSource:0}: Error finding container 3fbd5e047f1078ca3dbb5b5843d8497b2108b5bd97af187f2a03d7aebbd58f03: Status 404 returned error can't find the container with id 3fbd5e047f1078ca3dbb5b5843d8497b2108b5bd97af187f2a03d7aebbd58f03
Jul 30 04:08:38.082334 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:38.082212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw" event={"ID":"699b6a27-7ae3-4dc0-bc64-07f49fa46983","Type":"ContainerStarted","Data":"1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f"}
Jul 30 04:08:38.082334 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:38.082295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw" event={"ID":"699b6a27-7ae3-4dc0-bc64-07f49fa46983","Type":"ContainerStarted","Data":"3fbd5e047f1078ca3dbb5b5843d8497b2108b5bd97af187f2a03d7aebbd58f03"}
Jul 30 04:08:38.098860 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:38.098802    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw" podStartSLOduration=2.7251016740000003 podStartE2EDuration="3.098782306s" totalImagesPullingTime="373.680632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:08:38.097022302 +0000 UTC m=+2025.051656965" watchObservedRunningTime="2026-07-30 04:08:38.098782306 +0000 UTC m=+2025.053417003"
Jul 30 04:08:39.088082 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:39.088047    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:45.099043 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:45.099012    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:08:49.090399 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:49.090370    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:08:49.090784 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:08:49.090442    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:09:15.917162 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:15.916341    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"]
Jul 30 04:09:15.917162 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:15.916976    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw" podUID="699b6a27-7ae3-4dc0-bc64-07f49fa46983" containerName="maas-api" containerID="cri-o://1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f" gracePeriod=30
Jul 30 04:09:15.965941 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:15.965906    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"]
Jul 30 04:09:15.966190 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:15.966163    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck" podUID="48c85f24-6019-4f24-904e-6bceff237f82" containerName="payload-processing" containerID="cri-o://10a762b45c336a9ec91a5ee81fedf1705618e85f48ea1b5d06be5a15d75724e3" gracePeriod=30
Jul 30 04:09:15.977154 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:15.977114    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"]
Jul 30 04:09:15.978303 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:15.977409    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn" podUID="3aacd380-f4b2-45ca-b7f1-f7e7890310f5" containerName="payload-pre-processing" containerID="cri-o://ab1c0b4e88c39354ee3c6257c6f933e0b39b5b383a9f77e1fd9c2040e7f2d4ae" gracePeriod=30
Jul 30 04:09:16.250957 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.250922    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:09:16.286438 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.286400    2560 generic.go:356] "Generic (PLEG): container finished" podID="3aacd380-f4b2-45ca-b7f1-f7e7890310f5" containerID="ab1c0b4e88c39354ee3c6257c6f933e0b39b5b383a9f77e1fd9c2040e7f2d4ae" exitCode=0
Jul 30 04:09:16.286582 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.286482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn" event={"ID":"3aacd380-f4b2-45ca-b7f1-f7e7890310f5","Type":"ContainerDied","Data":"ab1c0b4e88c39354ee3c6257c6f933e0b39b5b383a9f77e1fd9c2040e7f2d4ae"}
Jul 30 04:09:16.288636 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.288607    2560 generic.go:356] "Generic (PLEG): container finished" podID="48c85f24-6019-4f24-904e-6bceff237f82" containerID="10a762b45c336a9ec91a5ee81fedf1705618e85f48ea1b5d06be5a15d75724e3" exitCode=0
Jul 30 04:09:16.288781 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.288655    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck" event={"ID":"48c85f24-6019-4f24-904e-6bceff237f82","Type":"ContainerDied","Data":"10a762b45c336a9ec91a5ee81fedf1705618e85f48ea1b5d06be5a15d75724e3"}
Jul 30 04:09:16.290321 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.290289    2560 generic.go:356] "Generic (PLEG): container finished" podID="699b6a27-7ae3-4dc0-bc64-07f49fa46983" containerID="1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f" exitCode=0
Jul 30 04:09:16.290420 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.290333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw" event={"ID":"699b6a27-7ae3-4dc0-bc64-07f49fa46983","Type":"ContainerDied","Data":"1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f"}
Jul 30 04:09:16.290420 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.290339    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"
Jul 30 04:09:16.290420 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.290379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw" event={"ID":"699b6a27-7ae3-4dc0-bc64-07f49fa46983","Type":"ContainerDied","Data":"3fbd5e047f1078ca3dbb5b5843d8497b2108b5bd97af187f2a03d7aebbd58f03"}
Jul 30 04:09:16.290420 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.290415    2560 scope.go:117] "RemoveContainer" containerID="1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f"
Jul 30 04:09:16.293313 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.293290    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:09:16.299785 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.299767    2560 scope.go:117] "RemoveContainer" containerID="1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f"
Jul 30 04:09:16.300033 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:09:16.300014    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f\": container with ID starting with 1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f not found: ID does not exist" containerID="1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f"
Jul 30 04:09:16.300103 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.300048    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f"} err="failed to get container status \"1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f\": rpc error: code = NotFound desc = could not find container \"1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f\": container with ID starting with 1ee90d2cb9c54d1d1e8f8c351ec95305c6b0276ed5df9967ee547d88f1532a2f not found: ID does not exist"
Jul 30 04:09:16.303129 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.303113    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:09:16.368709 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.368672    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c85f24-6019-4f24-904e-6bceff237f82-plugins-config-volume\") pod \"48c85f24-6019-4f24-904e-6bceff237f82\" (UID: \"48c85f24-6019-4f24-904e-6bceff237f82\") "
Jul 30 04:09:16.368941 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.368739    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rjgb\" (UniqueName: \"kubernetes.io/projected/48c85f24-6019-4f24-904e-6bceff237f82-kube-api-access-7rjgb\") pod \"48c85f24-6019-4f24-904e-6bceff237f82\" (UID: \"48c85f24-6019-4f24-904e-6bceff237f82\") "
Jul 30 04:09:16.368941 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.368770    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-plugins-config-volume\") pod \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\" (UID: \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\") "
Jul 30 04:09:16.368941 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.368793    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48vwq\" (UniqueName: \"kubernetes.io/projected/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-kube-api-access-48vwq\") pod \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\" (UID: \"3aacd380-f4b2-45ca-b7f1-f7e7890310f5\") "
Jul 30 04:09:16.369200 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.369167    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c85f24-6019-4f24-904e-6bceff237f82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48c85f24-6019-4f24-904e-6bceff237f82" (UID: "48c85f24-6019-4f24-904e-6bceff237f82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:09:16.369306 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.369177    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls\") pod \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") "
Jul 30 04:09:16.369306 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.369218    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3aacd380-f4b2-45ca-b7f1-f7e7890310f5" (UID: "3aacd380-f4b2-45ca-b7f1-f7e7890310f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:09:16.369421 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.369375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxnp5\" (UniqueName: \"kubernetes.io/projected/699b6a27-7ae3-4dc0-bc64-07f49fa46983-kube-api-access-dxnp5\") pod \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") "
Jul 30 04:09:16.369481 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.369466    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b6a27-7ae3-4dc0-bc64-07f49fa46983-openshift-service-ca\") pod \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\" (UID: \"699b6a27-7ae3-4dc0-bc64-07f49fa46983\") "
Jul 30 04:09:16.369819 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.369798    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c85f24-6019-4f24-904e-6bceff237f82-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:16.370113 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.370085    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699b6a27-7ae3-4dc0-bc64-07f49fa46983-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "699b6a27-7ae3-4dc0-bc64-07f49fa46983" (UID: "699b6a27-7ae3-4dc0-bc64-07f49fa46983"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:09:16.370211 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.370195    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:16.371711 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.371677    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-kube-api-access-48vwq" (OuterVolumeSpecName: "kube-api-access-48vwq") pod "3aacd380-f4b2-45ca-b7f1-f7e7890310f5" (UID: "3aacd380-f4b2-45ca-b7f1-f7e7890310f5"). InnerVolumeSpecName "kube-api-access-48vwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:09:16.371711 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.371688    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c85f24-6019-4f24-904e-6bceff237f82-kube-api-access-7rjgb" (OuterVolumeSpecName: "kube-api-access-7rjgb") pod "48c85f24-6019-4f24-904e-6bceff237f82" (UID: "48c85f24-6019-4f24-904e-6bceff237f82"). InnerVolumeSpecName "kube-api-access-7rjgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:09:16.372142 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.372119    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699b6a27-7ae3-4dc0-bc64-07f49fa46983-kube-api-access-dxnp5" (OuterVolumeSpecName: "kube-api-access-dxnp5") pod "699b6a27-7ae3-4dc0-bc64-07f49fa46983" (UID: "699b6a27-7ae3-4dc0-bc64-07f49fa46983"). InnerVolumeSpecName "kube-api-access-dxnp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:09:16.372425 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.372395    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "699b6a27-7ae3-4dc0-bc64-07f49fa46983" (UID: "699b6a27-7ae3-4dc0-bc64-07f49fa46983"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:09:16.471278 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.471157    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxnp5\" (UniqueName: \"kubernetes.io/projected/699b6a27-7ae3-4dc0-bc64-07f49fa46983-kube-api-access-dxnp5\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:16.471278 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.471191    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699b6a27-7ae3-4dc0-bc64-07f49fa46983-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:16.471278 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.471201    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rjgb\" (UniqueName: \"kubernetes.io/projected/48c85f24-6019-4f24-904e-6bceff237f82-kube-api-access-7rjgb\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:16.471278 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.471210    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48vwq\" (UniqueName: \"kubernetes.io/projected/3aacd380-f4b2-45ca-b7f1-f7e7890310f5-kube-api-access-48vwq\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:16.471278 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.471220    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699b6a27-7ae3-4dc0-bc64-07f49fa46983-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:16.612850 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.612790    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"]
Jul 30 04:09:16.614808 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:16.614782    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e1a4694c-67cc945856-9sklw"]
Jul 30 04:09:17.297506 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.297408    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"
Jul 30 04:09:17.297506 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.297424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck" event={"ID":"48c85f24-6019-4f24-904e-6bceff237f82","Type":"ContainerDied","Data":"eab67251d40b3bef59c1b7ad477bca60bdaa5210e8d8b2567eb1b40d65478794"}
Jul 30 04:09:17.297506 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.297492    2560 scope.go:117] "RemoveContainer" containerID="10a762b45c336a9ec91a5ee81fedf1705618e85f48ea1b5d06be5a15d75724e3"
Jul 30 04:09:17.299844 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.299807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn" event={"ID":"3aacd380-f4b2-45ca-b7f1-f7e7890310f5","Type":"ContainerDied","Data":"626153bcb472acbef2e6c626e88443a16a83691f7cbe75490b9f7ff091a4e704"}
Jul 30 04:09:17.299844 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.299839    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"
Jul 30 04:09:17.307788 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.307761    2560 scope.go:117] "RemoveContainer" containerID="ab1c0b4e88c39354ee3c6257c6f933e0b39b5b383a9f77e1fd9c2040e7f2d4ae"
Jul 30 04:09:17.322479 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.322448    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"]
Jul 30 04:09:17.325053 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.325028    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e1a4694c-675d6c8bdd-jnnzn"]
Jul 30 04:09:17.333989 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.333963    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"]
Jul 30 04:09:17.335793 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.335767    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1a4694c-75c94c6c5c-5mjck"]
Jul 30 04:09:17.579602 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.579518    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aacd380-f4b2-45ca-b7f1-f7e7890310f5" path="/var/lib/kubelet/pods/3aacd380-f4b2-45ca-b7f1-f7e7890310f5/volumes"
Jul 30 04:09:17.580073 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.580054    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c85f24-6019-4f24-904e-6bceff237f82" path="/var/lib/kubelet/pods/48c85f24-6019-4f24-904e-6bceff237f82/volumes"
Jul 30 04:09:17.580608 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:17.580590    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699b6a27-7ae3-4dc0-bc64-07f49fa46983" path="/var/lib/kubelet/pods/699b6a27-7ae3-4dc0-bc64-07f49fa46983/volumes"
Jul 30 04:09:20.630276 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.630215    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"]
Jul 30 04:09:20.630832 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.630803    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="699b6a27-7ae3-4dc0-bc64-07f49fa46983" containerName="maas-api"
Jul 30 04:09:20.630832 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.630829    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="699b6a27-7ae3-4dc0-bc64-07f49fa46983" containerName="maas-api"
Jul 30 04:09:20.630998 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.630856    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aacd380-f4b2-45ca-b7f1-f7e7890310f5" containerName="payload-pre-processing"
Jul 30 04:09:20.630998 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.630865    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aacd380-f4b2-45ca-b7f1-f7e7890310f5" containerName="payload-pre-processing"
Jul 30 04:09:20.630998 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.630889    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c85f24-6019-4f24-904e-6bceff237f82" containerName="payload-processing"
Jul 30 04:09:20.630998 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.630898    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c85f24-6019-4f24-904e-6bceff237f82" containerName="payload-processing"
Jul 30 04:09:20.631139 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.631016    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="699b6a27-7ae3-4dc0-bc64-07f49fa46983" containerName="maas-api"
Jul 30 04:09:20.631139 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.631030    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c85f24-6019-4f24-904e-6bceff237f82" containerName="payload-processing"
Jul 30 04:09:20.631139 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.631042    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aacd380-f4b2-45ca-b7f1-f7e7890310f5" containerName="payload-pre-processing"
Jul 30 04:09:20.635826 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.635800    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:20.637808 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.637576    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1d4857d1-serving-cert\""
Jul 30 04:09:20.644753 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.644732    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"]
Jul 30 04:09:20.710728 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.710688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:20.710728 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.710730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a600b6a-f374-42f1-a671-df9beed1bb8c-openshift-service-ca\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:20.710997 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.710763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69ps\" (UniqueName: \"kubernetes.io/projected/8a600b6a-f374-42f1-a671-df9beed1bb8c-kube-api-access-q69ps\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:20.812122 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.812073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:20.812344 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.812134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a600b6a-f374-42f1-a671-df9beed1bb8c-openshift-service-ca\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:20.812344 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.812170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69ps\" (UniqueName: \"kubernetes.io/projected/8a600b6a-f374-42f1-a671-df9beed1bb8c-kube-api-access-q69ps\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:20.812344 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:09:20.812180    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-serving-cert: secret "maas-api-e2e-ait-1d4857d1-serving-cert" not found
Jul 30 04:09:20.812344 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:09:20.812289    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls podName:8a600b6a-f374-42f1-a671-df9beed1bb8c nodeName:}" failed. No retries permitted until 2026-07-30 04:09:21.312265457 +0000 UTC m=+2068.266900120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls") pod "maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp" (UID: "8a600b6a-f374-42f1-a671-df9beed1bb8c") : secret "maas-api-e2e-ait-1d4857d1-serving-cert" not found
Jul 30 04:09:20.812882 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.812865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a600b6a-f374-42f1-a671-df9beed1bb8c-openshift-service-ca\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:20.820001 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:20.819979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69ps\" (UniqueName: \"kubernetes.io/projected/8a600b6a-f374-42f1-a671-df9beed1bb8c-kube-api-access-q69ps\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:21.238948 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.238912    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"]
Jul 30 04:09:21.245388 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.245195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:21.248177 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.248148    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1d4857d1\""
Jul 30 04:09:21.248548 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.248519    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1d4857d1-dockercfg-s82q7\""
Jul 30 04:09:21.250911 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.250887    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"]
Jul 30 04:09:21.285929 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.285894    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"]
Jul 30 04:09:21.289491 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.289473    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:21.298970 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.298941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"]
Jul 30 04:09:21.317656 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.317627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:21.317816 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.317664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc\" (UID: \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\") " pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:21.317816 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.317694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvlt\" (UniqueName: \"kubernetes.io/projected/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-kube-api-access-4gvlt\") pod \"payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc\" (UID: \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\") " pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:21.320670 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.320645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls\") pod \"maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:21.419172 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.419125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc\" (UID: \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\") " pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:21.419414 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.419187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvlt\" (UniqueName: \"kubernetes.io/projected/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-kube-api-access-4gvlt\") pod \"payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc\" (UID: \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\") " pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:21.419414 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.419252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5fm7\" (UniqueName: \"kubernetes.io/projected/e14642d8-caca-431f-81ff-d7c34b176ee0-kube-api-access-g5fm7\") pod \"payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9\" (UID: \"e14642d8-caca-431f-81ff-d7c34b176ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:21.419414 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.419344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14642d8-caca-431f-81ff-d7c34b176ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9\" (UID: \"e14642d8-caca-431f-81ff-d7c34b176ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:21.419964 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.419941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc\" (UID: \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\") " pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:21.426855 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.426825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvlt\" (UniqueName: \"kubernetes.io/projected/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-kube-api-access-4gvlt\") pod \"payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc\" (UID: \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\") " pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:21.520852 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.520760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5fm7\" (UniqueName: \"kubernetes.io/projected/e14642d8-caca-431f-81ff-d7c34b176ee0-kube-api-access-g5fm7\") pod \"payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9\" (UID: \"e14642d8-caca-431f-81ff-d7c34b176ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:21.520852 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.520846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14642d8-caca-431f-81ff-d7c34b176ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9\" (UID: \"e14642d8-caca-431f-81ff-d7c34b176ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:21.521680 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.521656    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14642d8-caca-431f-81ff-d7c34b176ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9\" (UID: \"e14642d8-caca-431f-81ff-d7c34b176ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:21.527650 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.527621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5fm7\" (UniqueName: \"kubernetes.io/projected/e14642d8-caca-431f-81ff-d7c34b176ee0-kube-api-access-g5fm7\") pod \"payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9\" (UID: \"e14642d8-caca-431f-81ff-d7c34b176ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:21.549531 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.549496    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:21.559905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.559855    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:21.601145 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.601114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:21.722982 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.722725    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"]
Jul 30 04:09:21.725475 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:09:21.725443    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a600b6a_f374_42f1_a671_df9beed1bb8c.slice/crio-f97ff1b89f24cc8d9b186f1c9db445d4416235a3fafae371cec2153c86737362 WatchSource:0}: Error finding container f97ff1b89f24cc8d9b186f1c9db445d4416235a3fafae371cec2153c86737362: Status 404 returned error can't find the container with id f97ff1b89f24cc8d9b186f1c9db445d4416235a3fafae371cec2153c86737362
Jul 30 04:09:21.748477 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.748449    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"]
Jul 30 04:09:21.752305 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:09:21.752225    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ffbb02_0ef1_47bc_b1b4_4c9b1321621e.slice/crio-f242dadf580310950718d9f953cbd937137f6b7a052e21f049042d25d81ed1c7 WatchSource:0}: Error finding container f242dadf580310950718d9f953cbd937137f6b7a052e21f049042d25d81ed1c7: Status 404 returned error can't find the container with id f242dadf580310950718d9f953cbd937137f6b7a052e21f049042d25d81ed1c7
Jul 30 04:09:21.780653 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:21.780609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"]
Jul 30 04:09:21.784531 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:09:21.784504    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14642d8_caca_431f_81ff_d7c34b176ee0.slice/crio-076271beb95da456e183e99977f687a7c6934653f4793c7f12b764e109635afd WatchSource:0}: Error finding container 076271beb95da456e183e99977f687a7c6934653f4793c7f12b764e109635afd: Status 404 returned error can't find the container with id 076271beb95da456e183e99977f687a7c6934653f4793c7f12b764e109635afd
Jul 30 04:09:22.329485 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.329444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp" event={"ID":"8a600b6a-f374-42f1-a671-df9beed1bb8c","Type":"ContainerStarted","Data":"f97ff1b89f24cc8d9b186f1c9db445d4416235a3fafae371cec2153c86737362"}
Jul 30 04:09:22.330965 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.330938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc" event={"ID":"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e","Type":"ContainerStarted","Data":"2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865"}
Jul 30 04:09:22.331087 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.330982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc" event={"ID":"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e","Type":"ContainerStarted","Data":"f242dadf580310950718d9f953cbd937137f6b7a052e21f049042d25d81ed1c7"}
Jul 30 04:09:22.331087 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.331025    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:22.332633 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.332602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9" event={"ID":"e14642d8-caca-431f-81ff-d7c34b176ee0","Type":"ContainerStarted","Data":"e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887"}
Jul 30 04:09:22.332745 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.332658    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:22.332745 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.332682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9" event={"ID":"e14642d8-caca-431f-81ff-d7c34b176ee0","Type":"ContainerStarted","Data":"076271beb95da456e183e99977f687a7c6934653f4793c7f12b764e109635afd"}
Jul 30 04:09:22.345262 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.345169    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc" podStartSLOduration=1.345154521 podStartE2EDuration="1.345154521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:09:22.343286279 +0000 UTC m=+2069.297920948" watchObservedRunningTime="2026-07-30 04:09:22.345154521 +0000 UTC m=+2069.299789193"
Jul 30 04:09:22.356750 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:22.356691    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9" podStartSLOduration=1.3566711759999999 podStartE2EDuration="1.356671176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:09:22.354941572 +0000 UTC m=+2069.309576243" watchObservedRunningTime="2026-07-30 04:09:22.356671176 +0000 UTC m=+2069.311305847"
Jul 30 04:09:23.339644 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:23.339601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp" event={"ID":"8a600b6a-f374-42f1-a671-df9beed1bb8c","Type":"ContainerStarted","Data":"dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec"}
Jul 30 04:09:23.340063 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:23.339824    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:23.353399 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:23.353346    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp" podStartSLOduration=2.707772462 podStartE2EDuration="3.353330713s" totalImagesPullingTime="645.558251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:09:23.35320013 +0000 UTC m=+2070.307834800" watchObservedRunningTime="2026-07-30 04:09:23.353330713 +0000 UTC m=+2070.307965381"
Jul 30 04:09:29.350486 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:29.350452    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:33.342085 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:33.342053    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:33.342519 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:33.342126    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:51.007134 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.007095    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"]
Jul 30 04:09:51.007622 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.007370    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp" podUID="8a600b6a-f374-42f1-a671-df9beed1bb8c" containerName="maas-api" containerID="cri-o://dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec" gracePeriod=30
Jul 30 04:09:51.050221 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.050184    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"]
Jul 30 04:09:51.050467 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.050436    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc" podUID="a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e" containerName="payload-processing" containerID="cri-o://2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865" gracePeriod=30
Jul 30 04:09:51.058812 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.058779    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"]
Jul 30 04:09:51.059123 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.059020    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9" podUID="e14642d8-caca-431f-81ff-d7c34b176ee0" containerName="payload-pre-processing" containerID="cri-o://e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887" gracePeriod=30
Jul 30 04:09:51.319089 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.319058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:51.355951 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.355927    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:51.377011 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.376986    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:51.410858 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.410829    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls\") pod \"8a600b6a-f374-42f1-a671-df9beed1bb8c\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") "
Jul 30 04:09:51.410858 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.410861    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5fm7\" (UniqueName: \"kubernetes.io/projected/e14642d8-caca-431f-81ff-d7c34b176ee0-kube-api-access-g5fm7\") pod \"e14642d8-caca-431f-81ff-d7c34b176ee0\" (UID: \"e14642d8-caca-431f-81ff-d7c34b176ee0\") "
Jul 30 04:09:51.411072 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.410904    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14642d8-caca-431f-81ff-d7c34b176ee0-plugins-config-volume\") pod \"e14642d8-caca-431f-81ff-d7c34b176ee0\" (UID: \"e14642d8-caca-431f-81ff-d7c34b176ee0\") "
Jul 30 04:09:51.411072 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.410945    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q69ps\" (UniqueName: \"kubernetes.io/projected/8a600b6a-f374-42f1-a671-df9beed1bb8c-kube-api-access-q69ps\") pod \"8a600b6a-f374-42f1-a671-df9beed1bb8c\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") "
Jul 30 04:09:51.411072 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.410971    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gvlt\" (UniqueName: \"kubernetes.io/projected/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-kube-api-access-4gvlt\") pod \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\" (UID: \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\") "
Jul 30 04:09:51.411072 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.410991    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-plugins-config-volume\") pod \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\" (UID: \"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e\") "
Jul 30 04:09:51.411311 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.411105    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a600b6a-f374-42f1-a671-df9beed1bb8c-openshift-service-ca\") pod \"8a600b6a-f374-42f1-a671-df9beed1bb8c\" (UID: \"8a600b6a-f374-42f1-a671-df9beed1bb8c\") "
Jul 30 04:09:51.411370 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.411336    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14642d8-caca-431f-81ff-d7c34b176ee0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e14642d8-caca-431f-81ff-d7c34b176ee0" (UID: "e14642d8-caca-431f-81ff-d7c34b176ee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:09:51.411436 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.411414    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e" (UID: "a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:09:51.411658 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.411637    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a600b6a-f374-42f1-a671-df9beed1bb8c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a600b6a-f374-42f1-a671-df9beed1bb8c" (UID: "8a600b6a-f374-42f1-a671-df9beed1bb8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:09:51.411721 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.411690    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14642d8-caca-431f-81ff-d7c34b176ee0-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:51.411721 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.411712    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:51.413216 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.413195    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14642d8-caca-431f-81ff-d7c34b176ee0-kube-api-access-g5fm7" (OuterVolumeSpecName: "kube-api-access-g5fm7") pod "e14642d8-caca-431f-81ff-d7c34b176ee0" (UID: "e14642d8-caca-431f-81ff-d7c34b176ee0"). InnerVolumeSpecName "kube-api-access-g5fm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:09:51.413312 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.413260    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a600b6a-f374-42f1-a671-df9beed1bb8c-kube-api-access-q69ps" (OuterVolumeSpecName: "kube-api-access-q69ps") pod "8a600b6a-f374-42f1-a671-df9beed1bb8c" (UID: "8a600b6a-f374-42f1-a671-df9beed1bb8c"). InnerVolumeSpecName "kube-api-access-q69ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:09:51.413477 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.413453    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a600b6a-f374-42f1-a671-df9beed1bb8c" (UID: "8a600b6a-f374-42f1-a671-df9beed1bb8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:09:51.413574 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.413456    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-kube-api-access-4gvlt" (OuterVolumeSpecName: "kube-api-access-4gvlt") pod "a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e" (UID: "a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e"). InnerVolumeSpecName "kube-api-access-4gvlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:09:51.490845 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.490812    2560 generic.go:356] "Generic (PLEG): container finished" podID="8a600b6a-f374-42f1-a671-df9beed1bb8c" containerID="dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec" exitCode=0
Jul 30 04:09:51.491044 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.490863    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"
Jul 30 04:09:51.491044 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.490902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp" event={"ID":"8a600b6a-f374-42f1-a671-df9beed1bb8c","Type":"ContainerDied","Data":"dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec"}
Jul 30 04:09:51.491044 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.490967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp" event={"ID":"8a600b6a-f374-42f1-a671-df9beed1bb8c","Type":"ContainerDied","Data":"f97ff1b89f24cc8d9b186f1c9db445d4416235a3fafae371cec2153c86737362"}
Jul 30 04:09:51.491044 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.490998    2560 scope.go:117] "RemoveContainer" containerID="dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec"
Jul 30 04:09:51.492325 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.492301    2560 generic.go:356] "Generic (PLEG): container finished" podID="a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e" containerID="2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865" exitCode=0
Jul 30 04:09:51.492396 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.492356    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"
Jul 30 04:09:51.492450 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.492390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc" event={"ID":"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e","Type":"ContainerDied","Data":"2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865"}
Jul 30 04:09:51.492494 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.492446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc" event={"ID":"a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e","Type":"ContainerDied","Data":"f242dadf580310950718d9f953cbd937137f6b7a052e21f049042d25d81ed1c7"}
Jul 30 04:09:51.493866 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.493834    2560 generic.go:356] "Generic (PLEG): container finished" podID="e14642d8-caca-431f-81ff-d7c34b176ee0" containerID="e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887" exitCode=0
Jul 30 04:09:51.493963 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.493867    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9" event={"ID":"e14642d8-caca-431f-81ff-d7c34b176ee0","Type":"ContainerDied","Data":"e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887"}
Jul 30 04:09:51.493963 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.493889    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"
Jul 30 04:09:51.493963 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.493920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9" event={"ID":"e14642d8-caca-431f-81ff-d7c34b176ee0","Type":"ContainerDied","Data":"076271beb95da456e183e99977f687a7c6934653f4793c7f12b764e109635afd"}
Jul 30 04:09:51.501956 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.501930    2560 scope.go:117] "RemoveContainer" containerID="dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec"
Jul 30 04:09:51.502292 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:09:51.502267    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec\": container with ID starting with dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec not found: ID does not exist" containerID="dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec"
Jul 30 04:09:51.502396 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.502303    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec"} err="failed to get container status \"dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec\": rpc error: code = NotFound desc = could not find container \"dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec\": container with ID starting with dc1ba9b414227df2e7b15324bd0d0a74f043ff084a80bba8995cb60cd902f8ec not found: ID does not exist"
Jul 30 04:09:51.502396 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.502328    2560 scope.go:117] "RemoveContainer" containerID="2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865"
Jul 30 04:09:51.510650 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.510623    2560 scope.go:117] "RemoveContainer" containerID="2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865"
Jul 30 04:09:51.510901 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:09:51.510879    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865\": container with ID starting with 2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865 not found: ID does not exist" containerID="2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865"
Jul 30 04:09:51.510946 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.510910    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865"} err="failed to get container status \"2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865\": rpc error: code = NotFound desc = could not find container \"2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865\": container with ID starting with 2f9e10aa82df1dfb54cb063374641171c1f48606555828246d0ccf100d722865 not found: ID does not exist"
Jul 30 04:09:51.510946 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.510927    2560 scope.go:117] "RemoveContainer" containerID="e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887"
Jul 30 04:09:51.512867 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.512842    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a600b6a-f374-42f1-a671-df9beed1bb8c-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:51.512867 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.512870    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5fm7\" (UniqueName: \"kubernetes.io/projected/e14642d8-caca-431f-81ff-d7c34b176ee0-kube-api-access-g5fm7\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:51.512867 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.512885    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q69ps\" (UniqueName: \"kubernetes.io/projected/8a600b6a-f374-42f1-a671-df9beed1bb8c-kube-api-access-q69ps\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:51.513126 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.512898    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvlt\" (UniqueName: \"kubernetes.io/projected/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e-kube-api-access-4gvlt\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:51.513126 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.512911    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a600b6a-f374-42f1-a671-df9beed1bb8c-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:09:51.514717 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.514696    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"]
Jul 30 04:09:51.518203 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.518181    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d4857d1-59b6f7f9f4-58pbp"]
Jul 30 04:09:51.519921 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.519905    2560 scope.go:117] "RemoveContainer" containerID="e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887"
Jul 30 04:09:51.520284 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:09:51.520207    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887\": container with ID starting with e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887 not found: ID does not exist" containerID="e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887"
Jul 30 04:09:51.520284 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.520262    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887"} err="failed to get container status \"e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887\": rpc error: code = NotFound desc = could not find container \"e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887\": container with ID starting with e37f16a4af120d65427cc2a5449926af221b84e156fc623d4460955c39519887 not found: ID does not exist"
Jul 30 04:09:51.525843 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.525819    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"]
Jul 30 04:09:51.530473 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.530449    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d4857d1-86f5897764-psrv9"]
Jul 30 04:09:51.540963 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.540929    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"]
Jul 30 04:09:51.547022 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.546997    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d4857d1-7bbf746c5b-qchlc"]
Jul 30 04:09:51.579429 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.579394    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a600b6a-f374-42f1-a671-df9beed1bb8c" path="/var/lib/kubelet/pods/8a600b6a-f374-42f1-a671-df9beed1bb8c/volumes"
Jul 30 04:09:51.579856 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.579836    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e" path="/var/lib/kubelet/pods/a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e/volumes"
Jul 30 04:09:51.580263 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:51.580219    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14642d8-caca-431f-81ff-d7c34b176ee0" path="/var/lib/kubelet/pods/e14642d8-caca-431f-81ff-d7c34b176ee0/volumes"
Jul 30 04:09:56.914966 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.914927    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"]
Jul 30 04:09:56.915462 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915445    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a600b6a-f374-42f1-a671-df9beed1bb8c" containerName="maas-api"
Jul 30 04:09:56.915462 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915463    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a600b6a-f374-42f1-a671-df9beed1bb8c" containerName="maas-api"
Jul 30 04:09:56.915605 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915484    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14642d8-caca-431f-81ff-d7c34b176ee0" containerName="payload-pre-processing"
Jul 30 04:09:56.915605 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915493    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14642d8-caca-431f-81ff-d7c34b176ee0" containerName="payload-pre-processing"
Jul 30 04:09:56.915605 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915511    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e" containerName="payload-processing"
Jul 30 04:09:56.915605 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915521    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e" containerName="payload-processing"
Jul 30 04:09:56.915816 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915633    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a600b6a-f374-42f1-a671-df9beed1bb8c" containerName="maas-api"
Jul 30 04:09:56.915816 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915648    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14642d8-caca-431f-81ff-d7c34b176ee0" containerName="payload-pre-processing"
Jul 30 04:09:56.915816 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.915661    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ffbb02-0ef1-47bc-b1b4-4c9b1321621e" containerName="payload-processing"
Jul 30 04:09:56.920972 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.920945    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:56.922873 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.922846    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3e3f5a98-serving-cert\""
Jul 30 04:09:56.928965 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:56.928941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"]
Jul 30 04:09:57.068168 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.068120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brhp\" (UniqueName: \"kubernetes.io/projected/82d7eaef-dacd-474e-98ff-1604a723eb9b-kube-api-access-9brhp\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.068381 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.068179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d7eaef-dacd-474e-98ff-1604a723eb9b-openshift-service-ca\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.068381 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.068308    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.169561 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.169473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.169721 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.169615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brhp\" (UniqueName: \"kubernetes.io/projected/82d7eaef-dacd-474e-98ff-1604a723eb9b-kube-api-access-9brhp\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.169721 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.169649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d7eaef-dacd-474e-98ff-1604a723eb9b-openshift-service-ca\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.169721 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:09:57.169659    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-serving-cert: secret "maas-api-e2e-derive-3e3f5a98-serving-cert" not found
Jul 30 04:09:57.169859 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:09:57.169754    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls podName:82d7eaef-dacd-474e-98ff-1604a723eb9b nodeName:}" failed. No retries permitted until 2026-07-30 04:09:57.669730745 +0000 UTC m=+2104.624365410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls") pod "maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx" (UID: "82d7eaef-dacd-474e-98ff-1604a723eb9b") : secret "maas-api-e2e-derive-3e3f5a98-serving-cert" not found
Jul 30 04:09:57.170485 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.170462    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d7eaef-dacd-474e-98ff-1604a723eb9b-openshift-service-ca\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.177651 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.177621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brhp\" (UniqueName: \"kubernetes.io/projected/82d7eaef-dacd-474e-98ff-1604a723eb9b-kube-api-access-9brhp\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.612623 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.612543    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"]
Jul 30 04:09:57.616316 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.616297    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:57.619073 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.619052    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3e3f5a98\""
Jul 30 04:09:57.619189 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.619052    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3e3f5a98-dockercfg-ts47j\""
Jul 30 04:09:57.637975 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.637942    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"]
Jul 30 04:09:57.671864 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.671833    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"]
Jul 30 04:09:57.675614 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.675593    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:57.675968 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.675946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a403a3-e968-4673-b33d-ebf39acce825-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg\" (UID: \"f1a403a3-e968-4673-b33d-ebf39acce825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:57.676021 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.675995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcfsn\" (UniqueName: \"kubernetes.io/projected/f1a403a3-e968-4673-b33d-ebf39acce825-kube-api-access-mcfsn\") pod \"payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg\" (UID: \"f1a403a3-e968-4673-b33d-ebf39acce825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:57.676139 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.676125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.678785 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.678754    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls\") pod \"maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.703670 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.703641    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"]
Jul 30 04:09:57.777036 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.776995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-plugins-config-volume\") pod \"payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g\" (UID: \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\") " pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:57.777260 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.777068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a403a3-e968-4673-b33d-ebf39acce825-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg\" (UID: \"f1a403a3-e968-4673-b33d-ebf39acce825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:57.777260 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.777092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlxcf\" (UniqueName: \"kubernetes.io/projected/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-kube-api-access-nlxcf\") pod \"payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g\" (UID: \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\") " pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:57.777260 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.777120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcfsn\" (UniqueName: \"kubernetes.io/projected/f1a403a3-e968-4673-b33d-ebf39acce825-kube-api-access-mcfsn\") pod \"payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg\" (UID: \"f1a403a3-e968-4673-b33d-ebf39acce825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:57.777830 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.777806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a403a3-e968-4673-b33d-ebf39acce825-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg\" (UID: \"f1a403a3-e968-4673-b33d-ebf39acce825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:57.799409 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.799383    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcfsn\" (UniqueName: \"kubernetes.io/projected/f1a403a3-e968-4673-b33d-ebf39acce825-kube-api-access-mcfsn\") pod \"payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg\" (UID: \"f1a403a3-e968-4673-b33d-ebf39acce825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:57.834442 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.834410    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:57.878464 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.878354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-plugins-config-volume\") pod \"payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g\" (UID: \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\") " pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:57.878464 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.878437    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlxcf\" (UniqueName: \"kubernetes.io/projected/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-kube-api-access-nlxcf\") pod \"payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g\" (UID: \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\") " pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:57.879058 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.879036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-plugins-config-volume\") pod \"payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g\" (UID: \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\") " pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:57.885920 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.885866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlxcf\" (UniqueName: \"kubernetes.io/projected/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-kube-api-access-nlxcf\") pod \"payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g\" (UID: \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\") " pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:57.945728 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.945425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:57.992077 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.992045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"]
Jul 30 04:09:57.993972 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:09:57.993944    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d7eaef_dacd_474e_98ff_1604a723eb9b.slice/crio-a44dce116843d48ddc2adcbc0a790d7e12c236eda1154873ac9ac28045e20d8b WatchSource:0}: Error finding container a44dce116843d48ddc2adcbc0a790d7e12c236eda1154873ac9ac28045e20d8b: Status 404 returned error can't find the container with id a44dce116843d48ddc2adcbc0a790d7e12c236eda1154873ac9ac28045e20d8b
Jul 30 04:09:57.995096 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:57.995070    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:58.104915 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.104888    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"]
Jul 30 04:09:58.108381 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:09:58.108355    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a403a3_e968_4673_b33d_ebf39acce825.slice/crio-6b337aaa8ec8bf5f29edc0e5dbf5635b16d2089c356f449025f39136dec1fac9 WatchSource:0}: Error finding container 6b337aaa8ec8bf5f29edc0e5dbf5635b16d2089c356f449025f39136dec1fac9: Status 404 returned error can't find the container with id 6b337aaa8ec8bf5f29edc0e5dbf5635b16d2089c356f449025f39136dec1fac9
Jul 30 04:09:58.355874 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.355495    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"]
Jul 30 04:09:58.357980 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:09:58.357951    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278e54c8_36cf_48ed_9b96_5a6d49f7dcf2.slice/crio-7a2645efa7ca6c8cb4d7f151d9ec0f631ba94ed0e8a7f2f030e4d12895604a4b WatchSource:0}: Error finding container 7a2645efa7ca6c8cb4d7f151d9ec0f631ba94ed0e8a7f2f030e4d12895604a4b: Status 404 returned error can't find the container with id 7a2645efa7ca6c8cb4d7f151d9ec0f631ba94ed0e8a7f2f030e4d12895604a4b
Jul 30 04:09:58.535436 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.535396    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g" event={"ID":"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2","Type":"ContainerStarted","Data":"458736b15936cc54ed13091fd7000505b0e2a0544fb37f69da8631ad93331531"}
Jul 30 04:09:58.535628 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.535453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g" event={"ID":"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2","Type":"ContainerStarted","Data":"7a2645efa7ca6c8cb4d7f151d9ec0f631ba94ed0e8a7f2f030e4d12895604a4b"}
Jul 30 04:09:58.535628 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.535578    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:09:58.536897 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.536866    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg" event={"ID":"f1a403a3-e968-4673-b33d-ebf39acce825","Type":"ContainerStarted","Data":"4e72414c43cf10f59c351c3594186933a67e5b52048d2d863d66d77f3c1932eb"}
Jul 30 04:09:58.537027 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.536918    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg" event={"ID":"f1a403a3-e968-4673-b33d-ebf39acce825","Type":"ContainerStarted","Data":"6b337aaa8ec8bf5f29edc0e5dbf5635b16d2089c356f449025f39136dec1fac9"}
Jul 30 04:09:58.537027 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.536946    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:09:58.538561 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.538538    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx" event={"ID":"82d7eaef-dacd-474e-98ff-1604a723eb9b","Type":"ContainerStarted","Data":"938301d9ec76448a2a13dc4c2e146dcf5fe669cc0bce0640317115887e520399"}
Jul 30 04:09:58.538667 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.538582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx" event={"ID":"82d7eaef-dacd-474e-98ff-1604a723eb9b","Type":"ContainerStarted","Data":"a44dce116843d48ddc2adcbc0a790d7e12c236eda1154873ac9ac28045e20d8b"}
Jul 30 04:09:58.538739 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.538683    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:09:58.548043 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.547988    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g" podStartSLOduration=1.547969597 podStartE2EDuration="1.547969597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:09:58.547410706 +0000 UTC m=+2105.502045377" watchObservedRunningTime="2026-07-30 04:09:58.547969597 +0000 UTC m=+2105.502604269"
Jul 30 04:09:58.561771 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.561716    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx" podStartSLOduration=2.166256164 podStartE2EDuration="2.561700106s" totalImagesPullingTime="395.443942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:09:58.560212412 +0000 UTC m=+2105.514847082" watchObservedRunningTime="2026-07-30 04:09:58.561700106 +0000 UTC m=+2105.516334776"
Jul 30 04:09:58.574732 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:09:58.574668    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg" podStartSLOduration=1.5746461809999999 podStartE2EDuration="1.574646181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:09:58.572265584 +0000 UTC m=+2105.526900252" watchObservedRunningTime="2026-07-30 04:09:58.574646181 +0000 UTC m=+2105.529280854"
Jul 30 04:10:04.553250 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:04.553206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:10:09.550350 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:09.550314    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:10:09.550763 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:09.550574    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:10:27.449736 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.449693    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"]
Jul 30 04:10:27.450178 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.449955    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx" podUID="82d7eaef-dacd-474e-98ff-1604a723eb9b" containerName="maas-api" containerID="cri-o://938301d9ec76448a2a13dc4c2e146dcf5fe669cc0bce0640317115887e520399" gracePeriod=30
Jul 30 04:10:27.489151 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.489109    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"]
Jul 30 04:10:27.489566 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.489521    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g" podUID="278e54c8-36cf-48ed-9b96-5a6d49f7dcf2" containerName="payload-processing" containerID="cri-o://458736b15936cc54ed13091fd7000505b0e2a0544fb37f69da8631ad93331531" gracePeriod=30
Jul 30 04:10:27.496017 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.495966    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"]
Jul 30 04:10:27.496250 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.496207    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg" podUID="f1a403a3-e968-4673-b33d-ebf39acce825" containerName="payload-pre-processing" containerID="cri-o://4e72414c43cf10f59c351c3594186933a67e5b52048d2d863d66d77f3c1932eb" gracePeriod=30
Jul 30 04:10:27.703394 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.703300    2560 generic.go:356] "Generic (PLEG): container finished" podID="f1a403a3-e968-4673-b33d-ebf39acce825" containerID="4e72414c43cf10f59c351c3594186933a67e5b52048d2d863d66d77f3c1932eb" exitCode=0
Jul 30 04:10:27.703394 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.703326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg" event={"ID":"f1a403a3-e968-4673-b33d-ebf39acce825","Type":"ContainerDied","Data":"4e72414c43cf10f59c351c3594186933a67e5b52048d2d863d66d77f3c1932eb"}
Jul 30 04:10:27.705227 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.705198    2560 generic.go:356] "Generic (PLEG): container finished" podID="82d7eaef-dacd-474e-98ff-1604a723eb9b" containerID="938301d9ec76448a2a13dc4c2e146dcf5fe669cc0bce0640317115887e520399" exitCode=0
Jul 30 04:10:27.705378 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.705343    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx" event={"ID":"82d7eaef-dacd-474e-98ff-1604a723eb9b","Type":"ContainerDied","Data":"938301d9ec76448a2a13dc4c2e146dcf5fe669cc0bce0640317115887e520399"}
Jul 30 04:10:27.707379 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.707290    2560 generic.go:356] "Generic (PLEG): container finished" podID="278e54c8-36cf-48ed-9b96-5a6d49f7dcf2" containerID="458736b15936cc54ed13091fd7000505b0e2a0544fb37f69da8631ad93331531" exitCode=0
Jul 30 04:10:27.707379 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.707349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g" event={"ID":"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2","Type":"ContainerDied","Data":"458736b15936cc54ed13091fd7000505b0e2a0544fb37f69da8631ad93331531"}
Jul 30 04:10:27.763518 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.763489    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:10:27.766879 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.766849    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:10:27.811921 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.811893    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:10:27.872669 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.872636    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls\") pod \"82d7eaef-dacd-474e-98ff-1604a723eb9b\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") "
Jul 30 04:10:27.872833 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.872696    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d7eaef-dacd-474e-98ff-1604a723eb9b-openshift-service-ca\") pod \"82d7eaef-dacd-474e-98ff-1604a723eb9b\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") "
Jul 30 04:10:27.872833 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.872732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlxcf\" (UniqueName: \"kubernetes.io/projected/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-kube-api-access-nlxcf\") pod \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\" (UID: \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\") "
Jul 30 04:10:27.872833 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.872762    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcfsn\" (UniqueName: \"kubernetes.io/projected/f1a403a3-e968-4673-b33d-ebf39acce825-kube-api-access-mcfsn\") pod \"f1a403a3-e968-4673-b33d-ebf39acce825\" (UID: \"f1a403a3-e968-4673-b33d-ebf39acce825\") "
Jul 30 04:10:27.873058 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.872859    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9brhp\" (UniqueName: \"kubernetes.io/projected/82d7eaef-dacd-474e-98ff-1604a723eb9b-kube-api-access-9brhp\") pod \"82d7eaef-dacd-474e-98ff-1604a723eb9b\" (UID: \"82d7eaef-dacd-474e-98ff-1604a723eb9b\") "
Jul 30 04:10:27.873058 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.872900    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a403a3-e968-4673-b33d-ebf39acce825-plugins-config-volume\") pod \"f1a403a3-e968-4673-b33d-ebf39acce825\" (UID: \"f1a403a3-e968-4673-b33d-ebf39acce825\") "
Jul 30 04:10:27.873058 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.872922    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-plugins-config-volume\") pod \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\" (UID: \"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2\") "
Jul 30 04:10:27.873267 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.873122    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d7eaef-dacd-474e-98ff-1604a723eb9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82d7eaef-dacd-474e-98ff-1604a723eb9b" (UID: "82d7eaef-dacd-474e-98ff-1604a723eb9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:10:27.873496 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.873445    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82d7eaef-dacd-474e-98ff-1604a723eb9b-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:10:27.873623 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.873575    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "278e54c8-36cf-48ed-9b96-5a6d49f7dcf2" (UID: "278e54c8-36cf-48ed-9b96-5a6d49f7dcf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:10:27.873728 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.873703    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a403a3-e968-4673-b33d-ebf39acce825-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1a403a3-e968-4673-b33d-ebf39acce825" (UID: "f1a403a3-e968-4673-b33d-ebf39acce825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:10:27.875004 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.874978    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a403a3-e968-4673-b33d-ebf39acce825-kube-api-access-mcfsn" (OuterVolumeSpecName: "kube-api-access-mcfsn") pod "f1a403a3-e968-4673-b33d-ebf39acce825" (UID: "f1a403a3-e968-4673-b33d-ebf39acce825"). InnerVolumeSpecName "kube-api-access-mcfsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:10:27.875004 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.875000    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82d7eaef-dacd-474e-98ff-1604a723eb9b" (UID: "82d7eaef-dacd-474e-98ff-1604a723eb9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:10:27.875156 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.875037    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-kube-api-access-nlxcf" (OuterVolumeSpecName: "kube-api-access-nlxcf") pod "278e54c8-36cf-48ed-9b96-5a6d49f7dcf2" (UID: "278e54c8-36cf-48ed-9b96-5a6d49f7dcf2"). InnerVolumeSpecName "kube-api-access-nlxcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:10:27.875521 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.875496    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d7eaef-dacd-474e-98ff-1604a723eb9b-kube-api-access-9brhp" (OuterVolumeSpecName: "kube-api-access-9brhp") pod "82d7eaef-dacd-474e-98ff-1604a723eb9b" (UID: "82d7eaef-dacd-474e-98ff-1604a723eb9b"). InnerVolumeSpecName "kube-api-access-9brhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:10:27.975084 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.974998    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brhp\" (UniqueName: \"kubernetes.io/projected/82d7eaef-dacd-474e-98ff-1604a723eb9b-kube-api-access-9brhp\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:10:27.975084 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.975036    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a403a3-e968-4673-b33d-ebf39acce825-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:10:27.975084 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.975049    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:10:27.975084 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.975062    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82d7eaef-dacd-474e-98ff-1604a723eb9b-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:10:27.975084 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.975075    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlxcf\" (UniqueName: \"kubernetes.io/projected/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2-kube-api-access-nlxcf\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:10:27.975374 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:27.975089    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcfsn\" (UniqueName: \"kubernetes.io/projected/f1a403a3-e968-4673-b33d-ebf39acce825-kube-api-access-mcfsn\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:10:28.713315 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.713287    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"
Jul 30 04:10:28.713315 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.713301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g" event={"ID":"278e54c8-36cf-48ed-9b96-5a6d49f7dcf2","Type":"ContainerDied","Data":"7a2645efa7ca6c8cb4d7f151d9ec0f631ba94ed0e8a7f2f030e4d12895604a4b"}
Jul 30 04:10:28.713782 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.713356    2560 scope.go:117] "RemoveContainer" containerID="458736b15936cc54ed13091fd7000505b0e2a0544fb37f69da8631ad93331531"
Jul 30 04:10:28.714916 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.714892    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"
Jul 30 04:10:28.715048 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.714895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg" event={"ID":"f1a403a3-e968-4673-b33d-ebf39acce825","Type":"ContainerDied","Data":"6b337aaa8ec8bf5f29edc0e5dbf5635b16d2089c356f449025f39136dec1fac9"}
Jul 30 04:10:28.716448 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.716380    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx" event={"ID":"82d7eaef-dacd-474e-98ff-1604a723eb9b","Type":"ContainerDied","Data":"a44dce116843d48ddc2adcbc0a790d7e12c236eda1154873ac9ac28045e20d8b"}
Jul 30 04:10:28.716448 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.716396    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"
Jul 30 04:10:28.724709 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.724688    2560 scope.go:117] "RemoveContainer" containerID="4e72414c43cf10f59c351c3594186933a67e5b52048d2d863d66d77f3c1932eb"
Jul 30 04:10:28.733689 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.733666    2560 scope.go:117] "RemoveContainer" containerID="938301d9ec76448a2a13dc4c2e146dcf5fe669cc0bce0640317115887e520399"
Jul 30 04:10:28.752540 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.752516    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"]
Jul 30 04:10:28.761434 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.761412    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3e3f5a98-bd9585784-hx5fg"]
Jul 30 04:10:28.788219 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.788181    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"]
Jul 30 04:10:28.793894 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.793869    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3e3f5a98-68f55885b7-7m5sx"]
Jul 30 04:10:28.820132 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.820091    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"]
Jul 30 04:10:28.825907 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:28.825204    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3e3f5a98-76684878bb-nhs4g"]
Jul 30 04:10:29.580061 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:29.580028    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="278e54c8-36cf-48ed-9b96-5a6d49f7dcf2" path="/var/lib/kubelet/pods/278e54c8-36cf-48ed-9b96-5a6d49f7dcf2/volumes"
Jul 30 04:10:29.580465 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:29.580449    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d7eaef-dacd-474e-98ff-1604a723eb9b" path="/var/lib/kubelet/pods/82d7eaef-dacd-474e-98ff-1604a723eb9b/volumes"
Jul 30 04:10:29.580768 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:10:29.580754    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a403a3-e968-4673-b33d-ebf39acce825" path="/var/lib/kubelet/pods/f1a403a3-e968-4673-b33d-ebf39acce825/volumes"
Jul 30 04:12:39.162312 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162216    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"]
Jul 30 04:12:39.162794 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162693    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d7eaef-dacd-474e-98ff-1604a723eb9b" containerName="maas-api"
Jul 30 04:12:39.162794 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162707    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d7eaef-dacd-474e-98ff-1604a723eb9b" containerName="maas-api"
Jul 30 04:12:39.162794 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162723    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278e54c8-36cf-48ed-9b96-5a6d49f7dcf2" containerName="payload-processing"
Jul 30 04:12:39.162794 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162729    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="278e54c8-36cf-48ed-9b96-5a6d49f7dcf2" containerName="payload-processing"
Jul 30 04:12:39.162794 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162741    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a403a3-e968-4673-b33d-ebf39acce825" containerName="payload-pre-processing"
Jul 30 04:12:39.162794 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162747    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a403a3-e968-4673-b33d-ebf39acce825" containerName="payload-pre-processing"
Jul 30 04:12:39.163012 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162800    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="278e54c8-36cf-48ed-9b96-5a6d49f7dcf2" containerName="payload-processing"
Jul 30 04:12:39.163012 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162812    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a403a3-e968-4673-b33d-ebf39acce825" containerName="payload-pre-processing"
Jul 30 04:12:39.163012 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.162818    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d7eaef-dacd-474e-98ff-1604a723eb9b" containerName="maas-api"
Jul 30 04:12:39.166360 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.166339    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.168262 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.168223    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-246dcb85-serving-cert\""
Jul 30 04:12:39.176903 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.176875    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"]
Jul 30 04:12:39.308430 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.308390    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwk6b\" (UniqueName: \"kubernetes.io/projected/4988c774-15f4-4a3e-b52d-fe10c454a972-kube-api-access-kwk6b\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.308606 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.308444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4988c774-15f4-4a3e-b52d-fe10c454a972-maas-api-tls\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.308606 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.308510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4988c774-15f4-4a3e-b52d-fe10c454a972-openshift-service-ca\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.409924 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.409893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwk6b\" (UniqueName: \"kubernetes.io/projected/4988c774-15f4-4a3e-b52d-fe10c454a972-kube-api-access-kwk6b\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.409924 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.409937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4988c774-15f4-4a3e-b52d-fe10c454a972-maas-api-tls\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.410159 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.409979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4988c774-15f4-4a3e-b52d-fe10c454a972-openshift-service-ca\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.410674 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.410651    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4988c774-15f4-4a3e-b52d-fe10c454a972-openshift-service-ca\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.412586 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.412529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4988c774-15f4-4a3e-b52d-fe10c454a972-maas-api-tls\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.416824 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.416804    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwk6b\" (UniqueName: \"kubernetes.io/projected/4988c774-15f4-4a3e-b52d-fe10c454a972-kube-api-access-kwk6b\") pod \"maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.483895 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.483862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:39.629310 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.629279    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"]
Jul 30 04:12:39.632355 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:12:39.632328    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4988c774_15f4_4a3e_b52d_fe10c454a972.slice/crio-5dd138325360c3755856764f5a39b5c8e7f87fc07604715353e2c79e59380fa3 WatchSource:0}: Error finding container 5dd138325360c3755856764f5a39b5c8e7f87fc07604715353e2c79e59380fa3: Status 404 returned error can't find the container with id 5dd138325360c3755856764f5a39b5c8e7f87fc07604715353e2c79e59380fa3
Jul 30 04:12:39.912928 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.912892    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"]
Jul 30 04:12:39.918398 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.918362    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:39.920193 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.920167    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-246dcb85-dockercfg-64drx\""
Jul 30 04:12:39.920193 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.920181    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-246dcb85\""
Jul 30 04:12:39.929918 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:39.929891    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"]
Jul 30 04:12:40.014378 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.014345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc304b-e961-44dd-b15f-8de199f971c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h\" (UID: \"84bc304b-e961-44dd-b15f-8de199f971c6\") " pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:40.014556 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.014399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqrx\" (UniqueName: \"kubernetes.io/projected/84bc304b-e961-44dd-b15f-8de199f971c6-kube-api-access-mgqrx\") pod \"payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h\" (UID: \"84bc304b-e961-44dd-b15f-8de199f971c6\") " pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:40.115380 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.115342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc304b-e961-44dd-b15f-8de199f971c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h\" (UID: \"84bc304b-e961-44dd-b15f-8de199f971c6\") " pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:40.115504 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.115441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqrx\" (UniqueName: \"kubernetes.io/projected/84bc304b-e961-44dd-b15f-8de199f971c6-kube-api-access-mgqrx\") pod \"payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h\" (UID: \"84bc304b-e961-44dd-b15f-8de199f971c6\") " pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:40.116837 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.116812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc304b-e961-44dd-b15f-8de199f971c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h\" (UID: \"84bc304b-e961-44dd-b15f-8de199f971c6\") " pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:40.123194 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.123171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqrx\" (UniqueName: \"kubernetes.io/projected/84bc304b-e961-44dd-b15f-8de199f971c6-kube-api-access-mgqrx\") pod \"payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h\" (UID: \"84bc304b-e961-44dd-b15f-8de199f971c6\") " pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:40.231477 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.231378    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:40.375705 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.375354    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"]
Jul 30 04:12:40.381251 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:12:40.381197    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bc304b_e961_44dd_b15f_8de199f971c6.slice/crio-52d7fb0084556255729bff4a83fefc0132b7949c7ed503da2cba0feee46ac4e5 WatchSource:0}: Error finding container 52d7fb0084556255729bff4a83fefc0132b7949c7ed503da2cba0feee46ac4e5: Status 404 returned error can't find the container with id 52d7fb0084556255729bff4a83fefc0132b7949c7ed503da2cba0feee46ac4e5
Jul 30 04:12:40.387679 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.387653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2" event={"ID":"4988c774-15f4-4a3e-b52d-fe10c454a972","Type":"ContainerStarted","Data":"c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b"}
Jul 30 04:12:40.387783 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.387703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2" event={"ID":"4988c774-15f4-4a3e-b52d-fe10c454a972","Type":"ContainerStarted","Data":"5dd138325360c3755856764f5a39b5c8e7f87fc07604715353e2c79e59380fa3"}
Jul 30 04:12:40.388803 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.388783    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:40.404252 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:40.404192    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2" podStartSLOduration=0.93875253 podStartE2EDuration="1.404175103s" totalImagesPullingTime="465.422573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:12:40.401263327 +0000 UTC m=+2267.355897997" watchObservedRunningTime="2026-07-30 04:12:40.404175103 +0000 UTC m=+2267.358809774"
Jul 30 04:12:41.395476 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:41.395444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h" event={"ID":"84bc304b-e961-44dd-b15f-8de199f971c6","Type":"ContainerStarted","Data":"601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516"}
Jul 30 04:12:41.395892 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:41.395522    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h" event={"ID":"84bc304b-e961-44dd-b15f-8de199f971c6","Type":"ContainerStarted","Data":"52d7fb0084556255729bff4a83fefc0132b7949c7ed503da2cba0feee46ac4e5"}
Jul 30 04:12:41.395892 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:41.395711    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:12:41.410596 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:41.410536    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h" podStartSLOduration=2.410514157 podStartE2EDuration="2.410514157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:12:41.409285853 +0000 UTC m=+2268.363920525" watchObservedRunningTime="2026-07-30 04:12:41.410514157 +0000 UTC m=+2268.365148830"
Jul 30 04:12:46.402010 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:46.401981    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:12:52.404061 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:12:52.404030    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:13:20.649755 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:20.649705    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"]
Jul 30 04:13:20.650368 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:20.649969    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2" podUID="4988c774-15f4-4a3e-b52d-fe10c454a972" containerName="maas-api" containerID="cri-o://c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b" gracePeriod=30
Jul 30 04:13:20.690721 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:20.690685    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"]
Jul 30 04:13:20.691016 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:20.690982    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h" podUID="84bc304b-e961-44dd-b15f-8de199f971c6" containerName="payload-processing" containerID="cri-o://601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516" gracePeriod=30
Jul 30 04:13:20.944463 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:20.944431    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:13:20.971409 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:20.971372    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:13:21.010646 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.010610    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4988c774-15f4-4a3e-b52d-fe10c454a972-openshift-service-ca\") pod \"4988c774-15f4-4a3e-b52d-fe10c454a972\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") "
Jul 30 04:13:21.010848 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.010698    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4988c774-15f4-4a3e-b52d-fe10c454a972-maas-api-tls\") pod \"4988c774-15f4-4a3e-b52d-fe10c454a972\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") "
Jul 30 04:13:21.010848 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.010791    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwk6b\" (UniqueName: \"kubernetes.io/projected/4988c774-15f4-4a3e-b52d-fe10c454a972-kube-api-access-kwk6b\") pod \"4988c774-15f4-4a3e-b52d-fe10c454a972\" (UID: \"4988c774-15f4-4a3e-b52d-fe10c454a972\") "
Jul 30 04:13:21.011080 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.011055    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4988c774-15f4-4a3e-b52d-fe10c454a972-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4988c774-15f4-4a3e-b52d-fe10c454a972" (UID: "4988c774-15f4-4a3e-b52d-fe10c454a972"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:13:21.012869 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.012837    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4988c774-15f4-4a3e-b52d-fe10c454a972-kube-api-access-kwk6b" (OuterVolumeSpecName: "kube-api-access-kwk6b") pod "4988c774-15f4-4a3e-b52d-fe10c454a972" (UID: "4988c774-15f4-4a3e-b52d-fe10c454a972"). InnerVolumeSpecName "kube-api-access-kwk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:13:21.012945 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.012843    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4988c774-15f4-4a3e-b52d-fe10c454a972-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4988c774-15f4-4a3e-b52d-fe10c454a972" (UID: "4988c774-15f4-4a3e-b52d-fe10c454a972"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:13:21.112329 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.112295    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgqrx\" (UniqueName: \"kubernetes.io/projected/84bc304b-e961-44dd-b15f-8de199f971c6-kube-api-access-mgqrx\") pod \"84bc304b-e961-44dd-b15f-8de199f971c6\" (UID: \"84bc304b-e961-44dd-b15f-8de199f971c6\") "
Jul 30 04:13:21.112492 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.112391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc304b-e961-44dd-b15f-8de199f971c6-plugins-config-volume\") pod \"84bc304b-e961-44dd-b15f-8de199f971c6\" (UID: \"84bc304b-e961-44dd-b15f-8de199f971c6\") "
Jul 30 04:13:21.112636 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.112622    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwk6b\" (UniqueName: \"kubernetes.io/projected/4988c774-15f4-4a3e-b52d-fe10c454a972-kube-api-access-kwk6b\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:13:21.112680 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.112639    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4988c774-15f4-4a3e-b52d-fe10c454a972-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:13:21.112680 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.112648    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4988c774-15f4-4a3e-b52d-fe10c454a972-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:13:21.112768 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.112739    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bc304b-e961-44dd-b15f-8de199f971c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84bc304b-e961-44dd-b15f-8de199f971c6" (UID: "84bc304b-e961-44dd-b15f-8de199f971c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:13:21.114442 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.114419    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bc304b-e961-44dd-b15f-8de199f971c6-kube-api-access-mgqrx" (OuterVolumeSpecName: "kube-api-access-mgqrx") pod "84bc304b-e961-44dd-b15f-8de199f971c6" (UID: "84bc304b-e961-44dd-b15f-8de199f971c6"). InnerVolumeSpecName "kube-api-access-mgqrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:13:21.213952 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.213857    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bc304b-e961-44dd-b15f-8de199f971c6-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:13:21.213952 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.213899    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgqrx\" (UniqueName: \"kubernetes.io/projected/84bc304b-e961-44dd-b15f-8de199f971c6-kube-api-access-mgqrx\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:13:21.610519 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.610481    2560 generic.go:356] "Generic (PLEG): container finished" podID="84bc304b-e961-44dd-b15f-8de199f971c6" containerID="601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516" exitCode=0
Jul 30 04:13:21.610712 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.610533    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"
Jul 30 04:13:21.610712 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.610572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h" event={"ID":"84bc304b-e961-44dd-b15f-8de199f971c6","Type":"ContainerDied","Data":"601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516"}
Jul 30 04:13:21.610712 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.610626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h" event={"ID":"84bc304b-e961-44dd-b15f-8de199f971c6","Type":"ContainerDied","Data":"52d7fb0084556255729bff4a83fefc0132b7949c7ed503da2cba0feee46ac4e5"}
Jul 30 04:13:21.610712 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.610659    2560 scope.go:117] "RemoveContainer" containerID="601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516"
Jul 30 04:13:21.612140 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.612114    2560 generic.go:356] "Generic (PLEG): container finished" podID="4988c774-15f4-4a3e-b52d-fe10c454a972" containerID="c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b" exitCode=0
Jul 30 04:13:21.612285 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.612167    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"
Jul 30 04:13:21.612285 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.612184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2" event={"ID":"4988c774-15f4-4a3e-b52d-fe10c454a972","Type":"ContainerDied","Data":"c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b"}
Jul 30 04:13:21.612285 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.612247    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2" event={"ID":"4988c774-15f4-4a3e-b52d-fe10c454a972","Type":"ContainerDied","Data":"5dd138325360c3755856764f5a39b5c8e7f87fc07604715353e2c79e59380fa3"}
Jul 30 04:13:21.620546 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.620519    2560 scope.go:117] "RemoveContainer" containerID="601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516"
Jul 30 04:13:21.620920 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:13:21.620885    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516\": container with ID starting with 601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516 not found: ID does not exist" containerID="601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516"
Jul 30 04:13:21.621039 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.620940    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516"} err="failed to get container status \"601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516\": rpc error: code = NotFound desc = could not find container \"601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516\": container with ID starting with 601d1b2e949346da6209459e6fa7cf283b94d77a2c18a743c23c61d5f0add516 not found: ID does not exist"
Jul 30 04:13:21.621039 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.620962    2560 scope.go:117] "RemoveContainer" containerID="c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b"
Jul 30 04:13:21.630555 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.630525    2560 scope.go:117] "RemoveContainer" containerID="c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b"
Jul 30 04:13:21.630793 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.630775    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"]
Jul 30 04:13:21.630909 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:13:21.630881    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b\": container with ID starting with c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b not found: ID does not exist" containerID="c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b"
Jul 30 04:13:21.630973 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.630918    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b"} err="failed to get container status \"c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b\": rpc error: code = NotFound desc = could not find container \"c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b\": container with ID starting with c2e474ed9b8727bbe3df0e8064df1ae8e4d52683c86b3b81eac9dc05540b7d5b not found: ID does not exist"
Jul 30 04:13:21.632751 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.632726    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-246dcb85-668d6944d9-w6gf2"]
Jul 30 04:13:21.640086 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.640058    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"]
Jul 30 04:13:21.643376 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:21.643353    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-246dcb85-699bdf9666-v9l2h"]
Jul 30 04:13:23.582294 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:23.582265    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4988c774-15f4-4a3e-b52d-fe10c454a972" path="/var/lib/kubelet/pods/4988c774-15f4-4a3e-b52d-fe10c454a972/volumes"
Jul 30 04:13:23.582683 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:13:23.582633    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84bc304b-e961-44dd-b15f-8de199f971c6" path="/var/lib/kubelet/pods/84bc304b-e961-44dd-b15f-8de199f971c6/volumes"
Jul 30 04:14:08.345785 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.345746    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"]
Jul 30 04:14:08.346635 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.345953    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96" podUID="755e8acc-c6ee-4577-b882-8ae74307929e" containerName="manager" containerID="cri-o://677c241d308d87cb04b8a39b6042a5bacedfa50b7bbac8f2b95110b1ee73ade1" gracePeriod=10
Jul 30 04:14:08.847582 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.847544    2560 generic.go:356] "Generic (PLEG): container finished" podID="755e8acc-c6ee-4577-b882-8ae74307929e" containerID="677c241d308d87cb04b8a39b6042a5bacedfa50b7bbac8f2b95110b1ee73ade1" exitCode=0
Jul 30 04:14:08.847749 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.847621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96" event={"ID":"755e8acc-c6ee-4577-b882-8ae74307929e","Type":"ContainerDied","Data":"677c241d308d87cb04b8a39b6042a5bacedfa50b7bbac8f2b95110b1ee73ade1"}
Jul 30 04:14:08.898095 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.898070    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 04:14:08.972869 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.972768    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/755e8acc-c6ee-4577-b882-8ae74307929e-extensions-socket-volume\") pod \"755e8acc-c6ee-4577-b882-8ae74307929e\" (UID: \"755e8acc-c6ee-4577-b882-8ae74307929e\") "
Jul 30 04:14:08.973045 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.972887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh586\" (UniqueName: \"kubernetes.io/projected/755e8acc-c6ee-4577-b882-8ae74307929e-kube-api-access-rh586\") pod \"755e8acc-c6ee-4577-b882-8ae74307929e\" (UID: \"755e8acc-c6ee-4577-b882-8ae74307929e\") "
Jul 30 04:14:08.973149 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.973123    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755e8acc-c6ee-4577-b882-8ae74307929e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "755e8acc-c6ee-4577-b882-8ae74307929e" (UID: "755e8acc-c6ee-4577-b882-8ae74307929e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:14:08.974896 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:08.974865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755e8acc-c6ee-4577-b882-8ae74307929e-kube-api-access-rh586" (OuterVolumeSpecName: "kube-api-access-rh586") pod "755e8acc-c6ee-4577-b882-8ae74307929e" (UID: "755e8acc-c6ee-4577-b882-8ae74307929e"). InnerVolumeSpecName "kube-api-access-rh586". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:14:09.074473 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:09.074431    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/755e8acc-c6ee-4577-b882-8ae74307929e-extensions-socket-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:14:09.074473 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:09.074466    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh586\" (UniqueName: \"kubernetes.io/projected/755e8acc-c6ee-4577-b882-8ae74307929e-kube-api-access-rh586\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:14:09.854281 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:09.854145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96" event={"ID":"755e8acc-c6ee-4577-b882-8ae74307929e","Type":"ContainerDied","Data":"a45894c6c5a50a6f943c0deaa9a0346d7c935abee51620333e9cff638cfe6ea3"}
Jul 30 04:14:09.854281 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:09.854176    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"
Jul 30 04:14:09.854281 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:09.854228    2560 scope.go:117] "RemoveContainer" containerID="677c241d308d87cb04b8a39b6042a5bacedfa50b7bbac8f2b95110b1ee73ade1"
Jul 30 04:14:09.877788 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:09.877761    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"]
Jul 30 04:14:09.883361 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:09.883333    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kj96"]
Jul 30 04:14:11.579025 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:11.578982    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="755e8acc-c6ee-4577-b882-8ae74307929e" path="/var/lib/kubelet/pods/755e8acc-c6ee-4577-b882-8ae74307929e/volumes"
Jul 30 04:14:53.755110 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:14:53.755075    2560 scope.go:117] "RemoveContainer" containerID="81df84dd66825fd272a8e4986857a7439a644ce638b47781f529bd559a6b3982"
Jul 30 04:15:00.181999 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.181959    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s"]
Jul 30 04:15:00.182495 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182460    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4988c774-15f4-4a3e-b52d-fe10c454a972" containerName="maas-api"
Jul 30 04:15:00.182495 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182481    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4988c774-15f4-4a3e-b52d-fe10c454a972" containerName="maas-api"
Jul 30 04:15:00.182495 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182491    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755e8acc-c6ee-4577-b882-8ae74307929e" containerName="manager"
Jul 30 04:15:00.182495 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182497    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="755e8acc-c6ee-4577-b882-8ae74307929e" containerName="manager"
Jul 30 04:15:00.182706 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182505    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84bc304b-e961-44dd-b15f-8de199f971c6" containerName="payload-processing"
Jul 30 04:15:00.182706 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182510    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="84bc304b-e961-44dd-b15f-8de199f971c6" containerName="payload-processing"
Jul 30 04:15:00.182706 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182586    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="755e8acc-c6ee-4577-b882-8ae74307929e" containerName="manager"
Jul 30 04:15:00.182706 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182594    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4988c774-15f4-4a3e-b52d-fe10c454a972" containerName="maas-api"
Jul 30 04:15:00.182706 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.182601    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="84bc304b-e961-44dd-b15f-8de199f971c6" containerName="payload-processing"
Jul 30 04:15:00.185708 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.185690    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s"
Jul 30 04:15:00.189424 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.189403    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:15:00.194945 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.194923    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s"
Jul 30 04:15:00.212704 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.212675    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s"]
Jul 30 04:15:00.345738 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:00.345706    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s"]
Jul 30 04:15:00.348731 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:15:00.348702    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcf3868_f8fd_4d7a_9e18_ed85fb429a91.slice/crio-a4aca61e4cfb5ffd7247ca0b0f63e61b23d2ef88d99d750463b0318e332b8a21 WatchSource:0}: Error finding container a4aca61e4cfb5ffd7247ca0b0f63e61b23d2ef88d99d750463b0318e332b8a21: Status 404 returned error can't find the container with id a4aca61e4cfb5ffd7247ca0b0f63e61b23d2ef88d99d750463b0318e332b8a21
Jul 30 04:15:01.116446 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:01.116407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s" event={"ID":"7bcf3868-f8fd-4d7a-9e18-ed85fb429a91","Type":"ContainerStarted","Data":"748ef79abd92fbc10a42d1c916644898fc60325d395bb808b218648305ff6014"}
Jul 30 04:15:01.116651 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:01.116464    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s" event={"ID":"7bcf3868-f8fd-4d7a-9e18-ed85fb429a91","Type":"ContainerStarted","Data":"a4aca61e4cfb5ffd7247ca0b0f63e61b23d2ef88d99d750463b0318e332b8a21"}
Jul 30 04:15:02.129743 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:02.129698    2560 generic.go:356] "Generic (PLEG): container finished" podID="7bcf3868-f8fd-4d7a-9e18-ed85fb429a91" containerID="748ef79abd92fbc10a42d1c916644898fc60325d395bb808b218648305ff6014" exitCode=0
Jul 30 04:15:02.130164 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:02.129778    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s" event={"ID":"7bcf3868-f8fd-4d7a-9e18-ed85fb429a91","Type":"ContainerDied","Data":"748ef79abd92fbc10a42d1c916644898fc60325d395bb808b218648305ff6014"}
Jul 30 04:15:03.269000 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:03.268961    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s"
Jul 30 04:15:04.139551 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:04.139520    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s"
Jul 30 04:15:04.139761 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:04.139546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756415-9hh6s" event={"ID":"7bcf3868-f8fd-4d7a-9e18-ed85fb429a91","Type":"ContainerDied","Data":"a4aca61e4cfb5ffd7247ca0b0f63e61b23d2ef88d99d750463b0318e332b8a21"}
Jul 30 04:15:04.139761 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:04.139601    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4aca61e4cfb5ffd7247ca0b0f63e61b23d2ef88d99d750463b0318e332b8a21"
Jul 30 04:15:23.952629 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:23.952550    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"]
Jul 30 04:15:23.953029 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:23.953010    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcf3868-f8fd-4d7a-9e18-ed85fb429a91" containerName="cleanup"
Jul 30 04:15:23.953100 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:23.953032    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcf3868-f8fd-4d7a-9e18-ed85fb429a91" containerName="cleanup"
Jul 30 04:15:23.953170 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:23.953158    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bcf3868-f8fd-4d7a-9e18-ed85fb429a91" containerName="cleanup"
Jul 30 04:15:23.956521 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:23.956502    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:23.960473 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:23.960452    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6v4xr\""
Jul 30 04:15:23.980055 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:23.980029    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"]
Jul 30 04:15:24.073480 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:24.073432    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnrzq\" (UniqueName: \"kubernetes.io/projected/877738e4-a61e-40ee-bef9-eb64606048c5-kube-api-access-lnrzq\") pod \"kuadrant-operator-controller-manager-86878c769f-rxm9m\" (UID: \"877738e4-a61e-40ee-bef9-eb64606048c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:24.073647 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:24.073488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/877738e4-a61e-40ee-bef9-eb64606048c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rxm9m\" (UID: \"877738e4-a61e-40ee-bef9-eb64606048c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:24.174067 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:24.174030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnrzq\" (UniqueName: \"kubernetes.io/projected/877738e4-a61e-40ee-bef9-eb64606048c5-kube-api-access-lnrzq\") pod \"kuadrant-operator-controller-manager-86878c769f-rxm9m\" (UID: \"877738e4-a61e-40ee-bef9-eb64606048c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:24.174299 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:24.174088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/877738e4-a61e-40ee-bef9-eb64606048c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rxm9m\" (UID: \"877738e4-a61e-40ee-bef9-eb64606048c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:24.174551 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:24.174528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/877738e4-a61e-40ee-bef9-eb64606048c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rxm9m\" (UID: \"877738e4-a61e-40ee-bef9-eb64606048c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:24.190311 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:24.190283    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnrzq\" (UniqueName: \"kubernetes.io/projected/877738e4-a61e-40ee-bef9-eb64606048c5-kube-api-access-lnrzq\") pod \"kuadrant-operator-controller-manager-86878c769f-rxm9m\" (UID: \"877738e4-a61e-40ee-bef9-eb64606048c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:24.267511 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:24.267415    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:24.416220 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:24.416167    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"]
Jul 30 04:15:24.418711 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:15:24.418679    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877738e4_a61e_40ee_bef9_eb64606048c5.slice/crio-404515938b91c790b11e2789410421e45d0b99554e17b50742e03d8996664ad8 WatchSource:0}: Error finding container 404515938b91c790b11e2789410421e45d0b99554e17b50742e03d8996664ad8: Status 404 returned error can't find the container with id 404515938b91c790b11e2789410421e45d0b99554e17b50742e03d8996664ad8
Jul 30 04:15:25.242925 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:25.242886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m" event={"ID":"877738e4-a61e-40ee-bef9-eb64606048c5","Type":"ContainerStarted","Data":"765bd9778200c800f516a048a24983f31c660c3158e38b205ac9e0d9102e9ec8"}
Jul 30 04:15:25.243571 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:25.242949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m" event={"ID":"877738e4-a61e-40ee-bef9-eb64606048c5","Type":"ContainerStarted","Data":"404515938b91c790b11e2789410421e45d0b99554e17b50742e03d8996664ad8"}
Jul 30 04:15:25.243571 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:25.243023    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:25.263412 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:25.263355    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m" podStartSLOduration=2.263337658 podStartE2EDuration="2.263337658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:15:25.261219888 +0000 UTC m=+2432.215854554" watchObservedRunningTime="2026-07-30 04:15:25.263337658 +0000 UTC m=+2432.217972342"
Jul 30 04:15:36.250227 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:36.250191    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rxm9m"
Jul 30 04:15:47.335363 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.335315    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"]
Jul 30 04:15:47.339642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.339615    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.342455 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.342421    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7278e753-serving-cert\""
Jul 30 04:15:47.350528 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.350497    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"]
Jul 30 04:15:47.390712 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.390656    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79z8\" (UniqueName: \"kubernetes.io/projected/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-kube-api-access-h79z8\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.390712 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.390713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-openshift-service-ca\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.390968 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.390814    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-maas-api-tls\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.491976 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.491934    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79z8\" (UniqueName: \"kubernetes.io/projected/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-kube-api-access-h79z8\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.491976 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.491980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-openshift-service-ca\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.492278 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.492039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-maas-api-tls\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.492916 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.492886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-openshift-service-ca\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.494719 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.494696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-maas-api-tls\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.499716 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.499695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79z8\" (UniqueName: \"kubernetes.io/projected/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-kube-api-access-h79z8\") pod \"maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.655053 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.655009    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:47.808075 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.808041    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"]
Jul 30 04:15:47.811084 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:15:47.811052    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751e9aae_a8d9_4445_87a7_ebdd0a023cdd.slice/crio-d58084bfb0031a4353cbdc2e3094a30f926b536cbd465b786000eb876d2cacfb WatchSource:0}: Error finding container d58084bfb0031a4353cbdc2e3094a30f926b536cbd465b786000eb876d2cacfb: Status 404 returned error can't find the container with id d58084bfb0031a4353cbdc2e3094a30f926b536cbd465b786000eb876d2cacfb
Jul 30 04:15:47.812455 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:47.812435    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:15:48.117860 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.117773    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"]
Jul 30 04:15:48.122626 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.122607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:48.125126 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.125101    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7278e753\""
Jul 30 04:15:48.134528 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.134504    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"]
Jul 30 04:15:48.200748 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.200703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rr6\" (UniqueName: \"kubernetes.io/projected/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-kube-api-access-p7rr6\") pod \"payload-processing-e2e-mt-7278e753-545667c47d-d8jjd\" (UID: \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\") " pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:48.200942 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.200753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-plugins-config-volume\") pod \"payload-processing-e2e-mt-7278e753-545667c47d-d8jjd\" (UID: \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\") " pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:48.301816 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.301779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rr6\" (UniqueName: \"kubernetes.io/projected/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-kube-api-access-p7rr6\") pod \"payload-processing-e2e-mt-7278e753-545667c47d-d8jjd\" (UID: \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\") " pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:48.301816 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.301823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-plugins-config-volume\") pod \"payload-processing-e2e-mt-7278e753-545667c47d-d8jjd\" (UID: \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\") " pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:48.302638 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.302611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-plugins-config-volume\") pod \"payload-processing-e2e-mt-7278e753-545667c47d-d8jjd\" (UID: \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\") " pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:48.311892 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.311864    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rr6\" (UniqueName: \"kubernetes.io/projected/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-kube-api-access-p7rr6\") pod \"payload-processing-e2e-mt-7278e753-545667c47d-d8jjd\" (UID: \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\") " pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:48.363115 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.363080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j" event={"ID":"751e9aae-a8d9-4445-87a7-ebdd0a023cdd","Type":"ContainerStarted","Data":"d58084bfb0031a4353cbdc2e3094a30f926b536cbd465b786000eb876d2cacfb"}
Jul 30 04:15:48.434932 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.434898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:48.580752 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:48.580728    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"]
Jul 30 04:15:48.583525 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:15:48.583492    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7eae9ae_b404_4ed7_926e_62ba9e64dd33.slice/crio-99a1c7be1dbffce069cc30e68e7368f3c6c26ac3a8c082aa60ed66e517be9b35 WatchSource:0}: Error finding container 99a1c7be1dbffce069cc30e68e7368f3c6c26ac3a8c082aa60ed66e517be9b35: Status 404 returned error can't find the container with id 99a1c7be1dbffce069cc30e68e7368f3c6c26ac3a8c082aa60ed66e517be9b35
Jul 30 04:15:49.370397 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:49.370356    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd" event={"ID":"a7eae9ae-b404-4ed7-926e-62ba9e64dd33","Type":"ContainerStarted","Data":"0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8"}
Jul 30 04:15:49.370873 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:49.370430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd" event={"ID":"a7eae9ae-b404-4ed7-926e-62ba9e64dd33","Type":"ContainerStarted","Data":"99a1c7be1dbffce069cc30e68e7368f3c6c26ac3a8c082aa60ed66e517be9b35"}
Jul 30 04:15:49.370873 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:49.370465    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:15:49.371976 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:49.371952    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j" event={"ID":"751e9aae-a8d9-4445-87a7-ebdd0a023cdd","Type":"ContainerStarted","Data":"e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f"}
Jul 30 04:15:49.372104 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:49.372066    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:15:49.385364 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:49.385310    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd" podStartSLOduration=1.385291565 podStartE2EDuration="1.385291565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:15:49.383563092 +0000 UTC m=+2456.338197763" watchObservedRunningTime="2026-07-30 04:15:49.385291565 +0000 UTC m=+2456.339926235"
Jul 30 04:15:49.399806 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:49.399748    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j" podStartSLOduration=1.8937143170000001 podStartE2EDuration="2.399730196s" totalImagesPullingTime="506.015879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:15:49.397327423 +0000 UTC m=+2456.351962095" watchObservedRunningTime="2026-07-30 04:15:49.399730196 +0000 UTC m=+2456.354364866"
Jul 30 04:15:55.383297 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:15:55.383258    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:16:00.379396 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:00.379359    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:16:55.601781 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:55.601744    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"]
Jul 30 04:16:55.605740 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:55.605718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"
Jul 30 04:16:55.607778 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:55.607751    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:16:55.612468 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:55.612431    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"]
Jul 30 04:16:55.655025 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:55.654977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91bd217-2a57-45c3-995b-f14a6658b44a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7278e753-qdhzj\" (UID: \"c91bd217-2a57-45c3-995b-f14a6658b44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"
Jul 30 04:16:55.756564 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:55.756528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91bd217-2a57-45c3-995b-f14a6658b44a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7278e753-qdhzj\" (UID: \"c91bd217-2a57-45c3-995b-f14a6658b44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"
Jul 30 04:16:55.757289 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:55.757267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91bd217-2a57-45c3-995b-f14a6658b44a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7278e753-qdhzj\" (UID: \"c91bd217-2a57-45c3-995b-f14a6658b44a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"
Jul 30 04:16:55.919274 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:55.919227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"
Jul 30 04:16:56.055291 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:56.055260    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"]
Jul 30 04:16:56.058177 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:16:56.058151    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91bd217_2a57_45c3_995b_f14a6658b44a.slice/crio-a990562b94ab75bb04f62db4e630e03e39649f1bc3b12b1459a79971172d280b WatchSource:0}: Error finding container a990562b94ab75bb04f62db4e630e03e39649f1bc3b12b1459a79971172d280b: Status 404 returned error can't find the container with id a990562b94ab75bb04f62db4e630e03e39649f1bc3b12b1459a79971172d280b
Jul 30 04:16:56.715672 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:56.715635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj" event={"ID":"c91bd217-2a57-45c3-995b-f14a6658b44a","Type":"ContainerStarted","Data":"bea8b9ec3ec3428273ed5e929e77120812196622b2b72c29badc236675e521e9"}
Jul 30 04:16:56.716156 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:56.715699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj" event={"ID":"c91bd217-2a57-45c3-995b-f14a6658b44a","Type":"ContainerStarted","Data":"a990562b94ab75bb04f62db4e630e03e39649f1bc3b12b1459a79971172d280b"}
Jul 30 04:16:56.730219 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:56.730158    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj" podStartSLOduration=1.730139745 podStartE2EDuration="1.730139745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:16:56.727654885 +0000 UTC m=+2523.682289557" watchObservedRunningTime="2026-07-30 04:16:56.730139745 +0000 UTC m=+2523.684774415"
Jul 30 04:16:57.721766 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:57.721732    2560 generic.go:356] "Generic (PLEG): container finished" podID="c91bd217-2a57-45c3-995b-f14a6658b44a" containerID="bea8b9ec3ec3428273ed5e929e77120812196622b2b72c29badc236675e521e9" exitCode=0
Jul 30 04:16:57.722163 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:57.721797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj" event={"ID":"c91bd217-2a57-45c3-995b-f14a6658b44a","Type":"ContainerDied","Data":"bea8b9ec3ec3428273ed5e929e77120812196622b2b72c29badc236675e521e9"}
Jul 30 04:16:58.867361 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:58.867328    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"
Jul 30 04:16:58.887404 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:58.887135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91bd217-2a57-45c3-995b-f14a6658b44a-maas-api-service-ca\") pod \"c91bd217-2a57-45c3-995b-f14a6658b44a\" (UID: \"c91bd217-2a57-45c3-995b-f14a6658b44a\") "
Jul 30 04:16:58.888101 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:58.888046    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91bd217-2a57-45c3-995b-f14a6658b44a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c91bd217-2a57-45c3-995b-f14a6658b44a" (UID: "c91bd217-2a57-45c3-995b-f14a6658b44a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:16:58.988604 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:58.988492    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91bd217-2a57-45c3-995b-f14a6658b44a-maas-api-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:16:59.733631 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:59.733603    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"
Jul 30 04:16:59.733860 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:59.733632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj" event={"ID":"c91bd217-2a57-45c3-995b-f14a6658b44a","Type":"ContainerDied","Data":"a990562b94ab75bb04f62db4e630e03e39649f1bc3b12b1459a79971172d280b"}
Jul 30 04:16:59.733860 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:16:59.733680    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a990562b94ab75bb04f62db4e630e03e39649f1bc3b12b1459a79971172d280b"
Jul 30 04:17:10.841713 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:10.841664    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"]
Jul 30 04:17:10.842529 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:10.842468    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j" podUID="751e9aae-a8d9-4445-87a7-ebdd0a023cdd" containerName="maas-api" containerID="cri-o://e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f" gracePeriod=30
Jul 30 04:17:10.895283 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:17:10.895220    2560 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751e9aae_a8d9_4445_87a7_ebdd0a023cdd.slice/crio-e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751e9aae_a8d9_4445_87a7_ebdd0a023cdd.slice/crio-conmon-e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f.scope\": RecentStats: unable to find data in memory cache]"
Jul 30 04:17:10.895437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:10.895301    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"]
Jul 30 04:17:10.895595 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:10.895540    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd" podUID="a7eae9ae-b404-4ed7-926e-62ba9e64dd33" containerName="payload-processing" containerID="cri-o://0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8" gracePeriod=30
Jul 30 04:17:11.032868 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.032802    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"]
Jul 30 04:17:11.036697 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.036652    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7278e753-qdhzj"]
Jul 30 04:17:11.170690 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.170658    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:17:11.172983 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.172957    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:17:11.317214 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317170    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7rr6\" (UniqueName: \"kubernetes.io/projected/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-kube-api-access-p7rr6\") pod \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\" (UID: \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\") "
Jul 30 04:17:11.317449 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317279    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-openshift-service-ca\") pod \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") "
Jul 30 04:17:11.317449 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317322    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-plugins-config-volume\") pod \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\" (UID: \"a7eae9ae-b404-4ed7-926e-62ba9e64dd33\") "
Jul 30 04:17:11.317449 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317345    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h79z8\" (UniqueName: \"kubernetes.io/projected/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-kube-api-access-h79z8\") pod \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") "
Jul 30 04:17:11.317638 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-maas-api-tls\") pod \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\" (UID: \"751e9aae-a8d9-4445-87a7-ebdd0a023cdd\") "
Jul 30 04:17:11.317751 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317720    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "751e9aae-a8d9-4445-87a7-ebdd0a023cdd" (UID: "751e9aae-a8d9-4445-87a7-ebdd0a023cdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:17:11.317829 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317760    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7eae9ae-b404-4ed7-926e-62ba9e64dd33" (UID: "a7eae9ae-b404-4ed7-926e-62ba9e64dd33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:17:11.317932 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317912    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:17:11.317981 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.317938    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:17:11.319554 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.319529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-kube-api-access-p7rr6" (OuterVolumeSpecName: "kube-api-access-p7rr6") pod "a7eae9ae-b404-4ed7-926e-62ba9e64dd33" (UID: "a7eae9ae-b404-4ed7-926e-62ba9e64dd33"). InnerVolumeSpecName "kube-api-access-p7rr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:17:11.319554 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.319534    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-kube-api-access-h79z8" (OuterVolumeSpecName: "kube-api-access-h79z8") pod "751e9aae-a8d9-4445-87a7-ebdd0a023cdd" (UID: "751e9aae-a8d9-4445-87a7-ebdd0a023cdd"). InnerVolumeSpecName "kube-api-access-h79z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:17:11.319671 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.319593    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "751e9aae-a8d9-4445-87a7-ebdd0a023cdd" (UID: "751e9aae-a8d9-4445-87a7-ebdd0a023cdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:17:11.418999 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.418957    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7rr6\" (UniqueName: \"kubernetes.io/projected/a7eae9ae-b404-4ed7-926e-62ba9e64dd33-kube-api-access-p7rr6\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:17:11.418999 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.418993    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79z8\" (UniqueName: \"kubernetes.io/projected/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-kube-api-access-h79z8\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:17:11.418999 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.419004    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751e9aae-a8d9-4445-87a7-ebdd0a023cdd-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:17:11.580532 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.580493    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c91bd217-2a57-45c3-995b-f14a6658b44a" path="/var/lib/kubelet/pods/c91bd217-2a57-45c3-995b-f14a6658b44a/volumes"
Jul 30 04:17:11.796651 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.796559    2560 generic.go:356] "Generic (PLEG): container finished" podID="751e9aae-a8d9-4445-87a7-ebdd0a023cdd" containerID="e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f" exitCode=0
Jul 30 04:17:11.796651 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.796610    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"
Jul 30 04:17:11.796884 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.796646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j" event={"ID":"751e9aae-a8d9-4445-87a7-ebdd0a023cdd","Type":"ContainerDied","Data":"e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f"}
Jul 30 04:17:11.796884 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.796707    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j" event={"ID":"751e9aae-a8d9-4445-87a7-ebdd0a023cdd","Type":"ContainerDied","Data":"d58084bfb0031a4353cbdc2e3094a30f926b536cbd465b786000eb876d2cacfb"}
Jul 30 04:17:11.796884 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.796736    2560 scope.go:117] "RemoveContainer" containerID="e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f"
Jul 30 04:17:11.798353 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.798324    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7eae9ae-b404-4ed7-926e-62ba9e64dd33" containerID="0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8" exitCode=0
Jul 30 04:17:11.798482 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.798376    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"
Jul 30 04:17:11.798482 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.798383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd" event={"ID":"a7eae9ae-b404-4ed7-926e-62ba9e64dd33","Type":"ContainerDied","Data":"0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8"}
Jul 30 04:17:11.798482 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.798426    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd" event={"ID":"a7eae9ae-b404-4ed7-926e-62ba9e64dd33","Type":"ContainerDied","Data":"99a1c7be1dbffce069cc30e68e7368f3c6c26ac3a8c082aa60ed66e517be9b35"}
Jul 30 04:17:11.806506 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.806477    2560 scope.go:117] "RemoveContainer" containerID="e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f"
Jul 30 04:17:11.806885 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:17:11.806864    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f\": container with ID starting with e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f not found: ID does not exist" containerID="e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f"
Jul 30 04:17:11.806995 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.806900    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f"} err="failed to get container status \"e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f\": rpc error: code = NotFound desc = could not find container \"e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f\": container with ID starting with e4215e1fc631c13ad50067d7fac9d4743e6702f61e66eafcd283f2f563af962f not found: ID does not exist"
Jul 30 04:17:11.806995 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.806925    2560 scope.go:117] "RemoveContainer" containerID="0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8"
Jul 30 04:17:11.816124 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.816099    2560 scope.go:117] "RemoveContainer" containerID="0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8"
Jul 30 04:17:11.816494 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:17:11.816468    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8\": container with ID starting with 0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8 not found: ID does not exist" containerID="0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8"
Jul 30 04:17:11.816581 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.816505    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8"} err="failed to get container status \"0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8\": rpc error: code = NotFound desc = could not find container \"0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8\": container with ID starting with 0d9cde31b9bb149d0fba2ee087e6d4b2b71ae6ad2ae10123c5071725204fbaf8 not found: ID does not exist"
Jul 30 04:17:11.816874 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.816851    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"]
Jul 30 04:17:11.820899 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.820872    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7278e753-545667c47d-d8jjd"]
Jul 30 04:17:11.829983 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.829941    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"]
Jul 30 04:17:11.832036 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:11.832012    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7278e753-5b9f446c89-xrr8j"]
Jul 30 04:17:13.580091 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:13.580057    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="751e9aae-a8d9-4445-87a7-ebdd0a023cdd" path="/var/lib/kubelet/pods/751e9aae-a8d9-4445-87a7-ebdd0a023cdd/volumes"
Jul 30 04:17:13.580530 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:17:13.580457    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7eae9ae-b404-4ed7-926e-62ba9e64dd33" path="/var/lib/kubelet/pods/a7eae9ae-b404-4ed7-926e-62ba9e64dd33/volumes"
Jul 30 04:19:22.568127 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.568089    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"]
Jul 30 04:19:22.569387 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569357    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c91bd217-2a57-45c3-995b-f14a6658b44a" containerName="revoke-keys"
Jul 30 04:19:22.569387 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569386    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c91bd217-2a57-45c3-995b-f14a6658b44a" containerName="revoke-keys"
Jul 30 04:19:22.569642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569408    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7eae9ae-b404-4ed7-926e-62ba9e64dd33" containerName="payload-processing"
Jul 30 04:19:22.569642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569418    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7eae9ae-b404-4ed7-926e-62ba9e64dd33" containerName="payload-processing"
Jul 30 04:19:22.569642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569451    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751e9aae-a8d9-4445-87a7-ebdd0a023cdd" containerName="maas-api"
Jul 30 04:19:22.569642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569460    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="751e9aae-a8d9-4445-87a7-ebdd0a023cdd" containerName="maas-api"
Jul 30 04:19:22.569642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569552    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="751e9aae-a8d9-4445-87a7-ebdd0a023cdd" containerName="maas-api"
Jul 30 04:19:22.569642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569564    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7eae9ae-b404-4ed7-926e-62ba9e64dd33" containerName="payload-processing"
Jul 30 04:19:22.569642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.569576    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c91bd217-2a57-45c3-995b-f14a6658b44a" containerName="revoke-keys"
Jul 30 04:19:22.573505 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.573444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.577520 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.577489    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8d492b-serving-cert\""
Jul 30 04:19:22.582882 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.582809    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"]
Jul 30 04:19:22.658465 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.658417    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8l2\" (UniqueName: \"kubernetes.io/projected/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-kube-api-access-6r8l2\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.658465 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.658464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.658685 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.658497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.759029 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.758985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8l2\" (UniqueName: \"kubernetes.io/projected/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-kube-api-access-6r8l2\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.759029 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.759037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.759352 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.759073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.759787 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.759767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.761529 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.761510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.766896 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.766868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8l2\" (UniqueName: \"kubernetes.io/projected/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-kube-api-access-6r8l2\") pod \"maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:22.890660 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:22.890627    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:23.035732 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.035700    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"]
Jul 30 04:19:23.038832 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:19:23.038800    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b9f02d_86db_4f3e_9acc_ffcb6c0e0420.slice/crio-05d5ad7eb704eafc5b35554f56451ee02ec7523527c855244c8255d89a72a020 WatchSource:0}: Error finding container 05d5ad7eb704eafc5b35554f56451ee02ec7523527c855244c8255d89a72a020: Status 404 returned error can't find the container with id 05d5ad7eb704eafc5b35554f56451ee02ec7523527c855244c8255d89a72a020
Jul 30 04:19:23.235751 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.235669    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"]
Jul 30 04:19:23.240363 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.240343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:23.243124 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.243099    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8d492b\""
Jul 30 04:19:23.243273 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.243187    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8d492b-dockercfg-lgsjk\""
Jul 30 04:19:23.253542 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.253515    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"]
Jul 30 04:19:23.366998 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.366958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpts\" (UniqueName: \"kubernetes.io/projected/351ebb3a-c656-4057-9516-536ecbbabe9e-kube-api-access-mfpts\") pod \"payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8\" (UID: \"351ebb3a-c656-4057-9516-536ecbbabe9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:23.367190 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.367009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351ebb3a-c656-4057-9516-536ecbbabe9e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8\" (UID: \"351ebb3a-c656-4057-9516-536ecbbabe9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:23.468012 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.467961    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpts\" (UniqueName: \"kubernetes.io/projected/351ebb3a-c656-4057-9516-536ecbbabe9e-kube-api-access-mfpts\") pod \"payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8\" (UID: \"351ebb3a-c656-4057-9516-536ecbbabe9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:23.468213 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.468033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351ebb3a-c656-4057-9516-536ecbbabe9e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8\" (UID: \"351ebb3a-c656-4057-9516-536ecbbabe9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:23.469001 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.468970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351ebb3a-c656-4057-9516-536ecbbabe9e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8\" (UID: \"351ebb3a-c656-4057-9516-536ecbbabe9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:23.476868 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.476834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpts\" (UniqueName: \"kubernetes.io/projected/351ebb3a-c656-4057-9516-536ecbbabe9e-kube-api-access-mfpts\") pod \"payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8\" (UID: \"351ebb3a-c656-4057-9516-536ecbbabe9e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:23.485930 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.485845    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb" event={"ID":"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420","Type":"ContainerStarted","Data":"05d5ad7eb704eafc5b35554f56451ee02ec7523527c855244c8255d89a72a020"}
Jul 30 04:19:23.551260 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.551203    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:23.700349 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:23.700208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"]
Jul 30 04:19:23.703107 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:19:23.703079    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351ebb3a_c656_4057_9516_536ecbbabe9e.slice/crio-e4221ea778286e3aaa94010763d14b26fcf1ff38d3ce54c80f9d93cf1ec91a7a WatchSource:0}: Error finding container e4221ea778286e3aaa94010763d14b26fcf1ff38d3ce54c80f9d93cf1ec91a7a: Status 404 returned error can't find the container with id e4221ea778286e3aaa94010763d14b26fcf1ff38d3ce54c80f9d93cf1ec91a7a
Jul 30 04:19:24.492752 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:24.492706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb" event={"ID":"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420","Type":"ContainerStarted","Data":"ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29"}
Jul 30 04:19:24.492971 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:24.492860    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:24.494469 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:24.494442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8" event={"ID":"351ebb3a-c656-4057-9516-536ecbbabe9e","Type":"ContainerStarted","Data":"cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec"}
Jul 30 04:19:24.494805 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:24.494483    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8" event={"ID":"351ebb3a-c656-4057-9516-536ecbbabe9e","Type":"ContainerStarted","Data":"e4221ea778286e3aaa94010763d14b26fcf1ff38d3ce54c80f9d93cf1ec91a7a"}
Jul 30 04:19:24.494805 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:24.494514    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:24.509250 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:24.509171    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb" podStartSLOduration=1.490026574 podStartE2EDuration="2.509154072s" totalImagesPullingTime="1.019127498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:19:24.507902249 +0000 UTC m=+2671.462536933" watchObservedRunningTime="2026-07-30 04:19:24.509154072 +0000 UTC m=+2671.463788807"
Jul 30 04:19:24.524966 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:24.524907    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8" podStartSLOduration=1.524889226 podStartE2EDuration="1.524889226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:19:24.522612933 +0000 UTC m=+2671.477247605" watchObservedRunningTime="2026-07-30 04:19:24.524889226 +0000 UTC m=+2671.479523897"
Jul 30 04:19:30.506381 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:30.506349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:19:35.502728 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:35.502697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:19:50.741916 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.741879    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"]
Jul 30 04:19:50.745527 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.745506    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:50.748300 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.748278    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5f61e5-serving-cert\""
Jul 30 04:19:50.757930 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.757902    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"]
Jul 30 04:19:50.820067 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.820024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:50.820267 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.820088    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlprf\" (UniqueName: \"kubernetes.io/projected/68b7777e-68f4-475c-849d-0bd02c80923c-kube-api-access-nlprf\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:50.820267 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.820208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b7777e-68f4-475c-849d-0bd02c80923c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:50.921871 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.921822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b7777e-68f4-475c-849d-0bd02c80923c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:50.922074 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.921895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:50.922074 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.921941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlprf\" (UniqueName: \"kubernetes.io/projected/68b7777e-68f4-475c-849d-0bd02c80923c-kube-api-access-nlprf\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:50.922074 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:19:50.922026    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-serving-cert: secret "maas-api-e2e-inf-b-5f61e5-serving-cert" not found
Jul 30 04:19:50.922281 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:19:50.922108    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls podName:68b7777e-68f4-475c-849d-0bd02c80923c nodeName:}" failed. No retries permitted until 2026-07-30 04:19:51.422090641 +0000 UTC m=+2698.376725290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls") pod "maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl" (UID: "68b7777e-68f4-475c-849d-0bd02c80923c") : secret "maas-api-e2e-inf-b-5f61e5-serving-cert" not found
Jul 30 04:19:50.922631 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.922605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b7777e-68f4-475c-849d-0bd02c80923c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:50.930876 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:50.930850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlprf\" (UniqueName: \"kubernetes.io/projected/68b7777e-68f4-475c-849d-0bd02c80923c-kube-api-access-nlprf\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:51.428087 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.428053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:51.430715 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.430692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:51.457455 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.457415    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"]
Jul 30 04:19:51.461245 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.461210    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:51.463133 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.463111    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5f61e5\""
Jul 30 04:19:51.473416 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.473390    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"]
Jul 30 04:19:51.492517 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.492483    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"]
Jul 30 04:19:51.495970 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.495949    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:51.497938 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.497911    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5f61e5-dockercfg-trk6t\""
Jul 30 04:19:51.511136 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.511111    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"]
Jul 30 04:19:51.528700 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.528664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvbr\" (UniqueName: \"kubernetes.io/projected/6dded42a-caff-4830-b2e6-1e61c74d1a56-kube-api-access-rsvbr\") pod \"payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22\" (UID: \"6dded42a-caff-4830-b2e6-1e61c74d1a56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:51.528859 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.528757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dded42a-caff-4830-b2e6-1e61c74d1a56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22\" (UID: \"6dded42a-caff-4830-b2e6-1e61c74d1a56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:51.630413 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.630380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda86ba-023f-437d-9912-997bdb4f23b2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz\" (UID: \"7dda86ba-023f-437d-9912-997bdb4f23b2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:51.630605 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.630431    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvbr\" (UniqueName: \"kubernetes.io/projected/6dded42a-caff-4830-b2e6-1e61c74d1a56-kube-api-access-rsvbr\") pod \"payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22\" (UID: \"6dded42a-caff-4830-b2e6-1e61c74d1a56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:51.630605 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.630507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dded42a-caff-4830-b2e6-1e61c74d1a56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22\" (UID: \"6dded42a-caff-4830-b2e6-1e61c74d1a56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:51.630605 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.630535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjz4p\" (UniqueName: \"kubernetes.io/projected/7dda86ba-023f-437d-9912-997bdb4f23b2-kube-api-access-jjz4p\") pod \"payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz\" (UID: \"7dda86ba-023f-437d-9912-997bdb4f23b2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:51.631221 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.631198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dded42a-caff-4830-b2e6-1e61c74d1a56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22\" (UID: \"6dded42a-caff-4830-b2e6-1e61c74d1a56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:51.639934 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.639904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvbr\" (UniqueName: \"kubernetes.io/projected/6dded42a-caff-4830-b2e6-1e61c74d1a56-kube-api-access-rsvbr\") pod \"payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22\" (UID: \"6dded42a-caff-4830-b2e6-1e61c74d1a56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:51.659841 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.659816    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:51.731704 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.731494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjz4p\" (UniqueName: \"kubernetes.io/projected/7dda86ba-023f-437d-9912-997bdb4f23b2-kube-api-access-jjz4p\") pod \"payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz\" (UID: \"7dda86ba-023f-437d-9912-997bdb4f23b2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:51.731704 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.731624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda86ba-023f-437d-9912-997bdb4f23b2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz\" (UID: \"7dda86ba-023f-437d-9912-997bdb4f23b2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:51.732389 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.732363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda86ba-023f-437d-9912-997bdb4f23b2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz\" (UID: \"7dda86ba-023f-437d-9912-997bdb4f23b2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:51.739779 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.739486    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjz4p\" (UniqueName: \"kubernetes.io/projected/7dda86ba-023f-437d-9912-997bdb4f23b2-kube-api-access-jjz4p\") pod \"payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz\" (UID: \"7dda86ba-023f-437d-9912-997bdb4f23b2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:51.776584 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.776546    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:51.804970 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.804943    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"]
Jul 30 04:19:51.806399 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.806378    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:51.808186 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:19:51.808163    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b7777e_68f4_475c_849d_0bd02c80923c.slice/crio-d52fb6858eadd1366a1d14b7d182cf94ccec67ab3bbdf33737559db7012d07e4 WatchSource:0}: Error finding container d52fb6858eadd1366a1d14b7d182cf94ccec67ab3bbdf33737559db7012d07e4: Status 404 returned error can't find the container with id d52fb6858eadd1366a1d14b7d182cf94ccec67ab3bbdf33737559db7012d07e4
Jul 30 04:19:51.946980 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.946941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"]
Jul 30 04:19:51.951262 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:19:51.951195    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dded42a_caff_4830_b2e6_1e61c74d1a56.slice/crio-beb319ad06327be36b79615a32a908a5cb67432edc033a5568e9810148b294a2 WatchSource:0}: Error finding container beb319ad06327be36b79615a32a908a5cb67432edc033a5568e9810148b294a2: Status 404 returned error can't find the container with id beb319ad06327be36b79615a32a908a5cb67432edc033a5568e9810148b294a2
Jul 30 04:19:51.989936 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:51.989795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"]
Jul 30 04:19:51.991957 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:19:51.991927    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dda86ba_023f_437d_9912_997bdb4f23b2.slice/crio-4eb43cbbc97321177ffe891a33a5b241811dda2e872ec851b9bb3f896af93ccc WatchSource:0}: Error finding container 4eb43cbbc97321177ffe891a33a5b241811dda2e872ec851b9bb3f896af93ccc: Status 404 returned error can't find the container with id 4eb43cbbc97321177ffe891a33a5b241811dda2e872ec851b9bb3f896af93ccc
Jul 30 04:19:52.642435 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.642389    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl" event={"ID":"68b7777e-68f4-475c-849d-0bd02c80923c","Type":"ContainerStarted","Data":"e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f"}
Jul 30 04:19:52.642686 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.642462    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:19:52.642686 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.642487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl" event={"ID":"68b7777e-68f4-475c-849d-0bd02c80923c","Type":"ContainerStarted","Data":"d52fb6858eadd1366a1d14b7d182cf94ccec67ab3bbdf33737559db7012d07e4"}
Jul 30 04:19:52.644179 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.644144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz" event={"ID":"7dda86ba-023f-437d-9912-997bdb4f23b2","Type":"ContainerStarted","Data":"f6d75602ee1f21e4354f5048e1f1fe305785aaabeaa42d1a0721efe2ef6c2d55"}
Jul 30 04:19:52.644341 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.644199    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz" event={"ID":"7dda86ba-023f-437d-9912-997bdb4f23b2","Type":"ContainerStarted","Data":"4eb43cbbc97321177ffe891a33a5b241811dda2e872ec851b9bb3f896af93ccc"}
Jul 30 04:19:52.644341 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.644222    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:19:52.645721 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.645697    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22" event={"ID":"6dded42a-caff-4830-b2e6-1e61c74d1a56","Type":"ContainerStarted","Data":"7e0e5c79aabd8591fa74a3ef5a16ef34cf52f66b4b3e15483ce565b86ff5720f"}
Jul 30 04:19:52.645836 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.645736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22" event={"ID":"6dded42a-caff-4830-b2e6-1e61c74d1a56","Type":"ContainerStarted","Data":"beb319ad06327be36b79615a32a908a5cb67432edc033a5568e9810148b294a2"}
Jul 30 04:19:52.645836 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.645754    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:19:52.660933 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.660883    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl" podStartSLOduration=2.346144071 podStartE2EDuration="2.660865035s" totalImagesPullingTime="314.720964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:19:52.659414433 +0000 UTC m=+2699.614049105" watchObservedRunningTime="2026-07-30 04:19:52.660865035 +0000 UTC m=+2699.615499705"
Jul 30 04:19:52.673913 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.673853    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz" podStartSLOduration=1.673819155 podStartE2EDuration="1.673819155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:19:52.673282107 +0000 UTC m=+2699.627916775" watchObservedRunningTime="2026-07-30 04:19:52.673819155 +0000 UTC m=+2699.628453827"
Jul 30 04:19:52.689725 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:52.689646    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22" podStartSLOduration=1.6896253909999999 podStartE2EDuration="1.689625391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:19:52.687448558 +0000 UTC m=+2699.642083232" watchObservedRunningTime="2026-07-30 04:19:52.689625391 +0000 UTC m=+2699.644260062"
Jul 30 04:19:58.656316 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:19:58.656285    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:20:03.653296 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:20:03.653245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:20:03.653874 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:20:03.653329    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:22:16.143486 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.143449    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"]
Jul 30 04:22:16.147530 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.147509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"
Jul 30 04:22:16.149868 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.149843    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:22:16.158533 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.158503    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"]
Jul 30 04:22:16.258886 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.258838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ab28b-77f4-4531-9542-1095a3b1a19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn\" (UID: \"9e1ab28b-77f4-4531-9542-1095a3b1a19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"
Jul 30 04:22:16.360508 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.360460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ab28b-77f4-4531-9542-1095a3b1a19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn\" (UID: \"9e1ab28b-77f4-4531-9542-1095a3b1a19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"
Jul 30 04:22:16.361152 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.361129    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ab28b-77f4-4531-9542-1095a3b1a19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn\" (UID: \"9e1ab28b-77f4-4531-9542-1095a3b1a19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"
Jul 30 04:22:16.459473 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.459383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"
Jul 30 04:22:16.606391 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:16.606328    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"]
Jul 30 04:22:16.608958 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:22:16.608924    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1ab28b_77f4_4531_9542_1095a3b1a19d.slice/crio-9d91a20b9fa9615f4b9406367e1732f1542d5c21064b70eb86d93b97cad40071 WatchSource:0}: Error finding container 9d91a20b9fa9615f4b9406367e1732f1542d5c21064b70eb86d93b97cad40071: Status 404 returned error can't find the container with id 9d91a20b9fa9615f4b9406367e1732f1542d5c21064b70eb86d93b97cad40071
Jul 30 04:22:17.418253 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:17.418192    2560 generic.go:356] "Generic (PLEG): container finished" podID="9e1ab28b-77f4-4531-9542-1095a3b1a19d" containerID="e10691e8ec307876b729ce614c6b77d400721aa7309a2c0cf13893cdf31fba12" exitCode=0
Jul 30 04:22:17.418662 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:17.418284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn" event={"ID":"9e1ab28b-77f4-4531-9542-1095a3b1a19d","Type":"ContainerDied","Data":"e10691e8ec307876b729ce614c6b77d400721aa7309a2c0cf13893cdf31fba12"}
Jul 30 04:22:17.418662 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:17.418350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn" event={"ID":"9e1ab28b-77f4-4531-9542-1095a3b1a19d","Type":"ContainerStarted","Data":"9d91a20b9fa9615f4b9406367e1732f1542d5c21064b70eb86d93b97cad40071"}
Jul 30 04:22:18.565273 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:18.565222    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"
Jul 30 04:22:18.683251 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:18.683149    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ab28b-77f4-4531-9542-1095a3b1a19d-maas-api-service-ca\") pod \"9e1ab28b-77f4-4531-9542-1095a3b1a19d\" (UID: \"9e1ab28b-77f4-4531-9542-1095a3b1a19d\") "
Jul 30 04:22:18.683630 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:18.683600    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1ab28b-77f4-4531-9542-1095a3b1a19d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e1ab28b-77f4-4531-9542-1095a3b1a19d" (UID: "9e1ab28b-77f4-4531-9542-1095a3b1a19d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:22:18.683814 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:18.683796    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1ab28b-77f4-4531-9542-1095a3b1a19d-maas-api-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:19.431905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:19.431869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn" event={"ID":"9e1ab28b-77f4-4531-9542-1095a3b1a19d","Type":"ContainerDied","Data":"9d91a20b9fa9615f4b9406367e1732f1542d5c21064b70eb86d93b97cad40071"}
Jul 30 04:22:19.432107 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:19.431920    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d91a20b9fa9615f4b9406367e1732f1542d5c21064b70eb86d93b97cad40071"
Jul 30 04:22:19.432107 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:19.431879    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"
Jul 30 04:22:26.793413 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:26.793381    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"]
Jul 30 04:22:26.793873 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:26.793635    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb" podUID="59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" containerName="maas-api" containerID="cri-o://ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29" gracePeriod=30
Jul 30 04:22:26.835470 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:26.835439    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"]
Jul 30 04:22:26.835717 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:26.835675    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8" podUID="351ebb3a-c656-4057-9516-536ecbbabe9e" containerName="payload-processing" containerID="cri-o://cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec" gracePeriod=30
Jul 30 04:22:26.993806 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:26.993720    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"]
Jul 30 04:22:26.996457 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:26.996425    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d492b-2hdhn"]
Jul 30 04:22:27.106601 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.106572    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:22:27.138011 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.137980    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:22:27.174774 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.174743    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-openshift-service-ca\") pod \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") "
Jul 30 04:22:27.174970 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.174895    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-maas-api-tls\") pod \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") "
Jul 30 04:22:27.174970 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.174947    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r8l2\" (UniqueName: \"kubernetes.io/projected/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-kube-api-access-6r8l2\") pod \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\" (UID: \"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420\") "
Jul 30 04:22:27.175222 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.175193    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" (UID: "59b9f02d-86db-4f3e-9acc-ffcb6c0e0420"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:22:27.177112 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.177081    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-kube-api-access-6r8l2" (OuterVolumeSpecName: "kube-api-access-6r8l2") pod "59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" (UID: "59b9f02d-86db-4f3e-9acc-ffcb6c0e0420"). InnerVolumeSpecName "kube-api-access-6r8l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:22:27.177280 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.177253    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" (UID: "59b9f02d-86db-4f3e-9acc-ffcb6c0e0420"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:22:27.275845 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.275808    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfpts\" (UniqueName: \"kubernetes.io/projected/351ebb3a-c656-4057-9516-536ecbbabe9e-kube-api-access-mfpts\") pod \"351ebb3a-c656-4057-9516-536ecbbabe9e\" (UID: \"351ebb3a-c656-4057-9516-536ecbbabe9e\") "
Jul 30 04:22:27.275845 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.275850    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351ebb3a-c656-4057-9516-536ecbbabe9e-plugins-config-volume\") pod \"351ebb3a-c656-4057-9516-536ecbbabe9e\" (UID: \"351ebb3a-c656-4057-9516-536ecbbabe9e\") "
Jul 30 04:22:27.276087 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.276032    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:27.276087 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.276042    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r8l2\" (UniqueName: \"kubernetes.io/projected/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-kube-api-access-6r8l2\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:27.276087 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.276052    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:27.276281 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.276223    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351ebb3a-c656-4057-9516-536ecbbabe9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "351ebb3a-c656-4057-9516-536ecbbabe9e" (UID: "351ebb3a-c656-4057-9516-536ecbbabe9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:22:27.278023 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.278001    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351ebb3a-c656-4057-9516-536ecbbabe9e-kube-api-access-mfpts" (OuterVolumeSpecName: "kube-api-access-mfpts") pod "351ebb3a-c656-4057-9516-536ecbbabe9e" (UID: "351ebb3a-c656-4057-9516-536ecbbabe9e"). InnerVolumeSpecName "kube-api-access-mfpts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:22:27.377685 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.377600    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfpts\" (UniqueName: \"kubernetes.io/projected/351ebb3a-c656-4057-9516-536ecbbabe9e-kube-api-access-mfpts\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:27.377685 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.377637    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351ebb3a-c656-4057-9516-536ecbbabe9e-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:27.485345 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.485302    2560 generic.go:356] "Generic (PLEG): container finished" podID="351ebb3a-c656-4057-9516-536ecbbabe9e" containerID="cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec" exitCode=0
Jul 30 04:22:27.485532 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.485364    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"
Jul 30 04:22:27.485532 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.485395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8" event={"ID":"351ebb3a-c656-4057-9516-536ecbbabe9e","Type":"ContainerDied","Data":"cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec"}
Jul 30 04:22:27.485532 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.485479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8" event={"ID":"351ebb3a-c656-4057-9516-536ecbbabe9e","Type":"ContainerDied","Data":"e4221ea778286e3aaa94010763d14b26fcf1ff38d3ce54c80f9d93cf1ec91a7a"}
Jul 30 04:22:27.485532 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.485512    2560 scope.go:117] "RemoveContainer" containerID="cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec"
Jul 30 04:22:27.487117 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.487077    2560 generic.go:356] "Generic (PLEG): container finished" podID="59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" containerID="ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29" exitCode=0
Jul 30 04:22:27.487272 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.487112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb" event={"ID":"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420","Type":"ContainerDied","Data":"ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29"}
Jul 30 04:22:27.487272 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.487154    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"
Jul 30 04:22:27.487272 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.487176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb" event={"ID":"59b9f02d-86db-4f3e-9acc-ffcb6c0e0420","Type":"ContainerDied","Data":"05d5ad7eb704eafc5b35554f56451ee02ec7523527c855244c8255d89a72a020"}
Jul 30 04:22:27.496034 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.496015    2560 scope.go:117] "RemoveContainer" containerID="cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec"
Jul 30 04:22:27.496350 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:22:27.496329    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec\": container with ID starting with cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec not found: ID does not exist" containerID="cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec"
Jul 30 04:22:27.496437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.496362    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec"} err="failed to get container status \"cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec\": rpc error: code = NotFound desc = could not find container \"cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec\": container with ID starting with cbe80036a2d08152205ef2a427807d84365eb8616f449220037e2eb221ff93ec not found: ID does not exist"
Jul 30 04:22:27.496437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.496384    2560 scope.go:117] "RemoveContainer" containerID="ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29"
Jul 30 04:22:27.504746 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.504726    2560 scope.go:117] "RemoveContainer" containerID="ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29"
Jul 30 04:22:27.505027 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:22:27.505007    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29\": container with ID starting with ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29 not found: ID does not exist" containerID="ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29"
Jul 30 04:22:27.505073 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.505034    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29"} err="failed to get container status \"ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29\": rpc error: code = NotFound desc = could not find container \"ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29\": container with ID starting with ea9a1d91c0e378a9d0a93a0f9a823ca67c6901357bdd172fb0fff089b33f3a29 not found: ID does not exist"
Jul 30 04:22:27.508162 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.508139    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"]
Jul 30 04:22:27.516539 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.516514    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d492b-bb8855464-rmng8"]
Jul 30 04:22:27.525911 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.525885    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"]
Jul 30 04:22:27.530535 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.530507    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d492b-8689b77d86-hvxgb"]
Jul 30 04:22:27.578810 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.578777    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351ebb3a-c656-4057-9516-536ecbbabe9e" path="/var/lib/kubelet/pods/351ebb3a-c656-4057-9516-536ecbbabe9e/volumes"
Jul 30 04:22:27.579185 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.579172    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" path="/var/lib/kubelet/pods/59b9f02d-86db-4f3e-9acc-ffcb6c0e0420/volumes"
Jul 30 04:22:27.579552 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:27.579538    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e1ab28b-77f4-4531-9542-1095a3b1a19d" path="/var/lib/kubelet/pods/9e1ab28b-77f4-4531-9542-1095a3b1a19d/volumes"
Jul 30 04:22:33.755738 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.755703    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"]
Jul 30 04:22:33.756339 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756319    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351ebb3a-c656-4057-9516-536ecbbabe9e" containerName="payload-processing"
Jul 30 04:22:33.756437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756340    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="351ebb3a-c656-4057-9516-536ecbbabe9e" containerName="payload-processing"
Jul 30 04:22:33.756437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756353    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" containerName="maas-api"
Jul 30 04:22:33.756437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756363    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" containerName="maas-api"
Jul 30 04:22:33.756437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756378    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e1ab28b-77f4-4531-9542-1095a3b1a19d" containerName="revoke-keys"
Jul 30 04:22:33.756437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756386    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e1ab28b-77f4-4531-9542-1095a3b1a19d" containerName="revoke-keys"
Jul 30 04:22:33.756687 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756527    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e1ab28b-77f4-4531-9542-1095a3b1a19d" containerName="revoke-keys"
Jul 30 04:22:33.756687 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756543    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b9f02d-86db-4f3e-9acc-ffcb6c0e0420" containerName="maas-api"
Jul 30 04:22:33.756687 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.756559    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="351ebb3a-c656-4057-9516-536ecbbabe9e" containerName="payload-processing"
Jul 30 04:22:33.759791 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.759768    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"
Jul 30 04:22:33.761665 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.761645    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:22:33.766440 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.766415    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"]
Jul 30 04:22:33.841770 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.841734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac51c1-2076-4333-b74c-03feca89ea52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx\" (UID: \"87ac51c1-2076-4333-b74c-03feca89ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"
Jul 30 04:22:33.942623 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.942584    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac51c1-2076-4333-b74c-03feca89ea52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx\" (UID: \"87ac51c1-2076-4333-b74c-03feca89ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"
Jul 30 04:22:33.943257 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:33.943213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac51c1-2076-4333-b74c-03feca89ea52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx\" (UID: \"87ac51c1-2076-4333-b74c-03feca89ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"
Jul 30 04:22:34.074041 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:34.073954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"
Jul 30 04:22:34.224210 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:34.224178    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"]
Jul 30 04:22:34.226348 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:22:34.226317    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ac51c1_2076_4333_b74c_03feca89ea52.slice/crio-87e22aa4adc01e02c0e185195474f6a43813e14cb2f964b8b82a78c3d9403c59 WatchSource:0}: Error finding container 87e22aa4adc01e02c0e185195474f6a43813e14cb2f964b8b82a78c3d9403c59: Status 404 returned error can't find the container with id 87e22aa4adc01e02c0e185195474f6a43813e14cb2f964b8b82a78c3d9403c59
Jul 30 04:22:34.527560 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:34.527524    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx" event={"ID":"87ac51c1-2076-4333-b74c-03feca89ea52","Type":"ContainerStarted","Data":"696ca936b52d29d68bd401d4bc03481d946d0e7d7ecdb11d194363363faaebe4"}
Jul 30 04:22:34.527746 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:34.527581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx" event={"ID":"87ac51c1-2076-4333-b74c-03feca89ea52","Type":"ContainerStarted","Data":"87e22aa4adc01e02c0e185195474f6a43813e14cb2f964b8b82a78c3d9403c59"}
Jul 30 04:22:34.545373 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:34.545319    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx" podStartSLOduration=1.545300715 podStartE2EDuration="1.545300715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:22:34.543116325 +0000 UTC m=+2861.497750997" watchObservedRunningTime="2026-07-30 04:22:34.545300715 +0000 UTC m=+2861.499935413"
Jul 30 04:22:35.534020 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:35.533982    2560 generic.go:356] "Generic (PLEG): container finished" podID="87ac51c1-2076-4333-b74c-03feca89ea52" containerID="696ca936b52d29d68bd401d4bc03481d946d0e7d7ecdb11d194363363faaebe4" exitCode=0
Jul 30 04:22:35.534470 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:35.534073    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx" event={"ID":"87ac51c1-2076-4333-b74c-03feca89ea52","Type":"ContainerDied","Data":"696ca936b52d29d68bd401d4bc03481d946d0e7d7ecdb11d194363363faaebe4"}
Jul 30 04:22:36.683669 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:36.683644    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"
Jul 30 04:22:36.772018 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:36.771979    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac51c1-2076-4333-b74c-03feca89ea52-maas-api-service-ca\") pod \"87ac51c1-2076-4333-b74c-03feca89ea52\" (UID: \"87ac51c1-2076-4333-b74c-03feca89ea52\") "
Jul 30 04:22:36.772417 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:36.772395    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ac51c1-2076-4333-b74c-03feca89ea52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87ac51c1-2076-4333-b74c-03feca89ea52" (UID: "87ac51c1-2076-4333-b74c-03feca89ea52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:22:36.873627 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:36.873529    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac51c1-2076-4333-b74c-03feca89ea52-maas-api-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:37.546176 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:37.546137    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"
Jul 30 04:22:37.546176 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:37.546163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx" event={"ID":"87ac51c1-2076-4333-b74c-03feca89ea52","Type":"ContainerDied","Data":"87e22aa4adc01e02c0e185195474f6a43813e14cb2f964b8b82a78c3d9403c59"}
Jul 30 04:22:37.546402 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:37.546214    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e22aa4adc01e02c0e185195474f6a43813e14cb2f964b8b82a78c3d9403c59"
Jul 30 04:22:44.220143 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.220100    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"]
Jul 30 04:22:44.220670 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.220445    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl" podUID="68b7777e-68f4-475c-849d-0bd02c80923c" containerName="maas-api" containerID="cri-o://e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f" gracePeriod=30
Jul 30 04:22:44.283430 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.283396    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"]
Jul 30 04:22:44.283693 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.283640    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz" podUID="7dda86ba-023f-437d-9912-997bdb4f23b2" containerName="payload-processing" containerID="cri-o://f6d75602ee1f21e4354f5048e1f1fe305785aaabeaa42d1a0721efe2ef6c2d55" gracePeriod=30
Jul 30 04:22:44.289419 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.289300    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"]
Jul 30 04:22:44.290097 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.289543    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22" podUID="6dded42a-caff-4830-b2e6-1e61c74d1a56" containerName="payload-pre-processing" containerID="cri-o://7e0e5c79aabd8591fa74a3ef5a16ef34cf52f66b4b3e15483ce565b86ff5720f" gracePeriod=30
Jul 30 04:22:44.439931 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.439893    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"]
Jul 30 04:22:44.443303 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.443268    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f61e5-kwdqx"]
Jul 30 04:22:44.560757 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.560721    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:22:44.588916 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.588880    2560 generic.go:356] "Generic (PLEG): container finished" podID="6dded42a-caff-4830-b2e6-1e61c74d1a56" containerID="7e0e5c79aabd8591fa74a3ef5a16ef34cf52f66b4b3e15483ce565b86ff5720f" exitCode=0
Jul 30 04:22:44.589060 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.588933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22" event={"ID":"6dded42a-caff-4830-b2e6-1e61c74d1a56","Type":"ContainerDied","Data":"7e0e5c79aabd8591fa74a3ef5a16ef34cf52f66b4b3e15483ce565b86ff5720f"}
Jul 30 04:22:44.589060 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.588985    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22" event={"ID":"6dded42a-caff-4830-b2e6-1e61c74d1a56","Type":"ContainerDied","Data":"beb319ad06327be36b79615a32a908a5cb67432edc033a5568e9810148b294a2"}
Jul 30 04:22:44.589060 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.589008    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beb319ad06327be36b79615a32a908a5cb67432edc033a5568e9810148b294a2"
Jul 30 04:22:44.590996 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.590970    2560 generic.go:356] "Generic (PLEG): container finished" podID="68b7777e-68f4-475c-849d-0bd02c80923c" containerID="e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f" exitCode=0
Jul 30 04:22:44.591135 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.591027    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"
Jul 30 04:22:44.591135 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.591100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl" event={"ID":"68b7777e-68f4-475c-849d-0bd02c80923c","Type":"ContainerDied","Data":"e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f"}
Jul 30 04:22:44.591276 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.591152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl" event={"ID":"68b7777e-68f4-475c-849d-0bd02c80923c","Type":"ContainerDied","Data":"d52fb6858eadd1366a1d14b7d182cf94ccec67ab3bbdf33737559db7012d07e4"}
Jul 30 04:22:44.591551 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.591330    2560 scope.go:117] "RemoveContainer" containerID="e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f"
Jul 30 04:22:44.592862 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.592722    2560 generic.go:356] "Generic (PLEG): container finished" podID="7dda86ba-023f-437d-9912-997bdb4f23b2" containerID="f6d75602ee1f21e4354f5048e1f1fe305785aaabeaa42d1a0721efe2ef6c2d55" exitCode=0
Jul 30 04:22:44.592862 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.592792    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz" event={"ID":"7dda86ba-023f-437d-9912-997bdb4f23b2","Type":"ContainerDied","Data":"f6d75602ee1f21e4354f5048e1f1fe305785aaabeaa42d1a0721efe2ef6c2d55"}
Jul 30 04:22:44.594348 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.594257    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:22:44.599642 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.599615    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:22:44.604340 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.604319    2560 scope.go:117] "RemoveContainer" containerID="e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f"
Jul 30 04:22:44.604689 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:22:44.604661    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f\": container with ID starting with e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f not found: ID does not exist" containerID="e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f"
Jul 30 04:22:44.604788 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.604692    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f"} err="failed to get container status \"e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f\": rpc error: code = NotFound desc = could not find container \"e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f\": container with ID starting with e4cc1c12f8c72b31a0ff3df80b83b9317b23957835565e868740b6f58bb25e2f not found: ID does not exist"
Jul 30 04:22:44.653704 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.653665    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda86ba-023f-437d-9912-997bdb4f23b2-plugins-config-volume\") pod \"7dda86ba-023f-437d-9912-997bdb4f23b2\" (UID: \"7dda86ba-023f-437d-9912-997bdb4f23b2\") "
Jul 30 04:22:44.653900 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.653729    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b7777e-68f4-475c-849d-0bd02c80923c-openshift-service-ca\") pod \"68b7777e-68f4-475c-849d-0bd02c80923c\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") "
Jul 30 04:22:44.653900 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.653755    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlprf\" (UniqueName: \"kubernetes.io/projected/68b7777e-68f4-475c-849d-0bd02c80923c-kube-api-access-nlprf\") pod \"68b7777e-68f4-475c-849d-0bd02c80923c\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") "
Jul 30 04:22:44.653900 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.653783    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjz4p\" (UniqueName: \"kubernetes.io/projected/7dda86ba-023f-437d-9912-997bdb4f23b2-kube-api-access-jjz4p\") pod \"7dda86ba-023f-437d-9912-997bdb4f23b2\" (UID: \"7dda86ba-023f-437d-9912-997bdb4f23b2\") "
Jul 30 04:22:44.653900 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.653810    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dded42a-caff-4830-b2e6-1e61c74d1a56-plugins-config-volume\") pod \"6dded42a-caff-4830-b2e6-1e61c74d1a56\" (UID: \"6dded42a-caff-4830-b2e6-1e61c74d1a56\") "
Jul 30 04:22:44.653900 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.653837    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls\") pod \"68b7777e-68f4-475c-849d-0bd02c80923c\" (UID: \"68b7777e-68f4-475c-849d-0bd02c80923c\") "
Jul 30 04:22:44.653900 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.653888    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsvbr\" (UniqueName: \"kubernetes.io/projected/6dded42a-caff-4830-b2e6-1e61c74d1a56-kube-api-access-rsvbr\") pod \"6dded42a-caff-4830-b2e6-1e61c74d1a56\" (UID: \"6dded42a-caff-4830-b2e6-1e61c74d1a56\") "
Jul 30 04:22:44.654688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.654144    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dda86ba-023f-437d-9912-997bdb4f23b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7dda86ba-023f-437d-9912-997bdb4f23b2" (UID: "7dda86ba-023f-437d-9912-997bdb4f23b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:22:44.654688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.654227    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dded42a-caff-4830-b2e6-1e61c74d1a56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dded42a-caff-4830-b2e6-1e61c74d1a56" (UID: "6dded42a-caff-4830-b2e6-1e61c74d1a56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:22:44.654688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.654259    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b7777e-68f4-475c-849d-0bd02c80923c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68b7777e-68f4-475c-849d-0bd02c80923c" (UID: "68b7777e-68f4-475c-849d-0bd02c80923c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:22:44.654688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.654439    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dda86ba-023f-437d-9912-997bdb4f23b2-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:44.654688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.654463    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b7777e-68f4-475c-849d-0bd02c80923c-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:44.654688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.654478    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dded42a-caff-4830-b2e6-1e61c74d1a56-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:44.656284 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.656251    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68b7777e-68f4-475c-849d-0bd02c80923c" (UID: "68b7777e-68f4-475c-849d-0bd02c80923c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:22:44.656405 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.656304    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dded42a-caff-4830-b2e6-1e61c74d1a56-kube-api-access-rsvbr" (OuterVolumeSpecName: "kube-api-access-rsvbr") pod "6dded42a-caff-4830-b2e6-1e61c74d1a56" (UID: "6dded42a-caff-4830-b2e6-1e61c74d1a56"). InnerVolumeSpecName "kube-api-access-rsvbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:22:44.656477 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.656459    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dda86ba-023f-437d-9912-997bdb4f23b2-kube-api-access-jjz4p" (OuterVolumeSpecName: "kube-api-access-jjz4p") pod "7dda86ba-023f-437d-9912-997bdb4f23b2" (UID: "7dda86ba-023f-437d-9912-997bdb4f23b2"). InnerVolumeSpecName "kube-api-access-jjz4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:22:44.656973 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.656957    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b7777e-68f4-475c-849d-0bd02c80923c-kube-api-access-nlprf" (OuterVolumeSpecName: "kube-api-access-nlprf") pod "68b7777e-68f4-475c-849d-0bd02c80923c" (UID: "68b7777e-68f4-475c-849d-0bd02c80923c"). InnerVolumeSpecName "kube-api-access-nlprf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:22:44.755913 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.755827    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlprf\" (UniqueName: \"kubernetes.io/projected/68b7777e-68f4-475c-849d-0bd02c80923c-kube-api-access-nlprf\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:44.755913 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.755858    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjz4p\" (UniqueName: \"kubernetes.io/projected/7dda86ba-023f-437d-9912-997bdb4f23b2-kube-api-access-jjz4p\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:44.755913 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.755868    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b7777e-68f4-475c-849d-0bd02c80923c-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:44.755913 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.755878    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsvbr\" (UniqueName: \"kubernetes.io/projected/6dded42a-caff-4830-b2e6-1e61c74d1a56-kube-api-access-rsvbr\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:22:44.920701 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.920662    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"]
Jul 30 04:22:44.926610 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:44.926583    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5f61e5-597d7cb594-p2nzl"]
Jul 30 04:22:45.581812 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.581728    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b7777e-68f4-475c-849d-0bd02c80923c" path="/var/lib/kubelet/pods/68b7777e-68f4-475c-849d-0bd02c80923c/volumes"
Jul 30 04:22:45.582171 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.582092    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ac51c1-2076-4333-b74c-03feca89ea52" path="/var/lib/kubelet/pods/87ac51c1-2076-4333-b74c-03feca89ea52/volumes"
Jul 30 04:22:45.600544 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.600517    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"
Jul 30 04:22:45.600709 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.600547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz" event={"ID":"7dda86ba-023f-437d-9912-997bdb4f23b2","Type":"ContainerDied","Data":"4eb43cbbc97321177ffe891a33a5b241811dda2e872ec851b9bb3f896af93ccc"}
Jul 30 04:22:45.600709 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.600619    2560 scope.go:117] "RemoveContainer" containerID="f6d75602ee1f21e4354f5048e1f1fe305785aaabeaa42d1a0721efe2ef6c2d55"
Jul 30 04:22:45.600912 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.600707    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"
Jul 30 04:22:45.618431 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.618397    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"]
Jul 30 04:22:45.621848 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.621820    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5f61e5-6cd9cf9d5b-msz22"]
Jul 30 04:22:45.631308 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.631277    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"]
Jul 30 04:22:45.633134 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:45.633111    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5f61e5-58fdfc76d4-ct6pz"]
Jul 30 04:22:47.579574 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:47.579531    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dded42a-caff-4830-b2e6-1e61c74d1a56" path="/var/lib/kubelet/pods/6dded42a-caff-4830-b2e6-1e61c74d1a56/volumes"
Jul 30 04:22:47.580213 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:47.580103    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dda86ba-023f-437d-9912-997bdb4f23b2" path="/var/lib/kubelet/pods/7dda86ba-023f-437d-9912-997bdb4f23b2/volumes"
Jul 30 04:22:55.715499 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.715456    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"]
Jul 30 04:22:55.716047 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716032    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dded42a-caff-4830-b2e6-1e61c74d1a56" containerName="payload-pre-processing"
Jul 30 04:22:55.716097 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716051    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dded42a-caff-4830-b2e6-1e61c74d1a56" containerName="payload-pre-processing"
Jul 30 04:22:55.716097 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716078    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b7777e-68f4-475c-849d-0bd02c80923c" containerName="maas-api"
Jul 30 04:22:55.716097 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716087    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b7777e-68f4-475c-849d-0bd02c80923c" containerName="maas-api"
Jul 30 04:22:55.716200 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716099    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ac51c1-2076-4333-b74c-03feca89ea52" containerName="revoke-keys"
Jul 30 04:22:55.716200 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716108    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ac51c1-2076-4333-b74c-03feca89ea52" containerName="revoke-keys"
Jul 30 04:22:55.716200 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716130    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dda86ba-023f-437d-9912-997bdb4f23b2" containerName="payload-processing"
Jul 30 04:22:55.716200 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716138    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dda86ba-023f-437d-9912-997bdb4f23b2" containerName="payload-processing"
Jul 30 04:22:55.716361 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716227    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dda86ba-023f-437d-9912-997bdb4f23b2" containerName="payload-processing"
Jul 30 04:22:55.716361 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716253    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ac51c1-2076-4333-b74c-03feca89ea52" containerName="revoke-keys"
Jul 30 04:22:55.716361 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716263    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dded42a-caff-4830-b2e6-1e61c74d1a56" containerName="payload-pre-processing"
Jul 30 04:22:55.716361 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.716274    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b7777e-68f4-475c-849d-0bd02c80923c" containerName="maas-api"
Jul 30 04:22:55.722084 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.722058    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.724197 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.724164    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9c7b98-serving-cert\""
Jul 30 04:22:55.728292 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.728258    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"]
Jul 30 04:22:55.757410 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.757368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.757576 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.757434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmb2\" (UniqueName: \"kubernetes.io/projected/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-kube-api-access-zvmb2\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.757576 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.757489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-maas-api-tls\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.858358 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.858323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmb2\" (UniqueName: \"kubernetes.io/projected/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-kube-api-access-zvmb2\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.858579 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.858379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-maas-api-tls\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.858579 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.858487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.859168 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.859147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.860975 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.860948    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-maas-api-tls\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:55.866210 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:55.866188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmb2\" (UniqueName: \"kubernetes.io/projected/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-kube-api-access-zvmb2\") pod \"maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:56.036382 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.036284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:56.183288 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.183253    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"]
Jul 30 04:22:56.186978 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:22:56.186943    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbabc4d_ea3e_4156_b0b9_9fba5aefa88b.slice/crio-92823fef0f23a394ae2087bd30953dde3d8beb4e347e4907d5e7b567c19cd9ad WatchSource:0}: Error finding container 92823fef0f23a394ae2087bd30953dde3d8beb4e347e4907d5e7b567c19cd9ad: Status 404 returned error can't find the container with id 92823fef0f23a394ae2087bd30953dde3d8beb4e347e4907d5e7b567c19cd9ad
Jul 30 04:22:56.188662 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.188640    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:22:56.417655 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.417619    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"]
Jul 30 04:22:56.422553 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.422531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:56.424635 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.424607    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9c7b98\""
Jul 30 04:22:56.424794 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.424652    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9c7b98-dockercfg-t5cnd\""
Jul 30 04:22:56.431975 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.431933    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"]
Jul 30 04:22:56.466058 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.466016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tx8z\" (UniqueName: \"kubernetes.io/projected/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-kube-api-access-8tx8z\") pod \"payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4\" (UID: \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\") " pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:56.466257 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.466130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4\" (UID: \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\") " pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:56.566784 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.566746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tx8z\" (UniqueName: \"kubernetes.io/projected/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-kube-api-access-8tx8z\") pod \"payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4\" (UID: \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\") " pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:56.566989 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.566824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4\" (UID: \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\") " pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:56.567707 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.567664    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4\" (UID: \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\") " pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:56.574345 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.574314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tx8z\" (UniqueName: \"kubernetes.io/projected/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-kube-api-access-8tx8z\") pod \"payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4\" (UID: \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\") " pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:56.666434 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.666405    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt" event={"ID":"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b","Type":"ContainerStarted","Data":"92823fef0f23a394ae2087bd30953dde3d8beb4e347e4907d5e7b567c19cd9ad"}
Jul 30 04:22:56.735957 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.735858    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:56.882968 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:56.882930    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"]
Jul 30 04:22:56.885461 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:22:56.885432    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0bc3d3_760f_4eb9_ba10_65bfccd1feba.slice/crio-4dd2d5d5883b220038d358514496be5dfbd16f08365074223b425fd3c05c8fc9 WatchSource:0}: Error finding container 4dd2d5d5883b220038d358514496be5dfbd16f08365074223b425fd3c05c8fc9: Status 404 returned error can't find the container with id 4dd2d5d5883b220038d358514496be5dfbd16f08365074223b425fd3c05c8fc9
Jul 30 04:22:57.675578 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:57.675540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt" event={"ID":"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b","Type":"ContainerStarted","Data":"ce3d115dd0001a7148d57d2b010c86971f28945f2be4f5b37d73333881a3eba7"}
Jul 30 04:22:57.675800 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:57.675612    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:22:57.677382 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:57.677353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4" event={"ID":"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba","Type":"ContainerStarted","Data":"29ef8521d8a66277d11f87afef05d9de1c75f2de16359e3e3bf44034bbfd68f0"}
Jul 30 04:22:57.677527 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:57.677400    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4" event={"ID":"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba","Type":"ContainerStarted","Data":"4dd2d5d5883b220038d358514496be5dfbd16f08365074223b425fd3c05c8fc9"}
Jul 30 04:22:57.677614 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:57.677598    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:22:57.690913 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:57.690856    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt" podStartSLOduration=2.271945514 podStartE2EDuration="2.690840105s" totalImagesPullingTime="418.894591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:22:57.689285207 +0000 UTC m=+2884.643919878" watchObservedRunningTime="2026-07-30 04:22:57.690840105 +0000 UTC m=+2884.645474776"
Jul 30 04:22:57.706046 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:22:57.705987    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4" podStartSLOduration=1.705970369 podStartE2EDuration="1.705970369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:22:57.703428771 +0000 UTC m=+2884.658063441" watchObservedRunningTime="2026-07-30 04:22:57.705970369 +0000 UTC m=+2884.660605040"
Jul 30 04:23:03.689181 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:03.689151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:23:08.685290 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:08.685259    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:23:23.741742 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.741703    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"]
Jul 30 04:23:23.745974 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.745935    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:23.747836 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.747815    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2a2d39-serving-cert\""
Jul 30 04:23:23.756052 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.755999    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"]
Jul 30 04:23:23.834736 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.834701    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a63843-3916-43be-a578-6aca95bc91e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:23.834919 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.834757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6bl\" (UniqueName: \"kubernetes.io/projected/82a63843-3916-43be-a578-6aca95bc91e1-kube-api-access-sv6bl\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:23.834919 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.834868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:23.936109 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.936065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a63843-3916-43be-a578-6aca95bc91e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:23.936339 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.936135    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6bl\" (UniqueName: \"kubernetes.io/projected/82a63843-3916-43be-a578-6aca95bc91e1-kube-api-access-sv6bl\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:23.936339 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.936204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:23.936460 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:23:23.936332    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-serving-cert: secret "maas-api-e2e-api-b-2a2d39-serving-cert" not found
Jul 30 04:23:23.936460 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:23:23.936404    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls podName:82a63843-3916-43be-a578-6aca95bc91e1 nodeName:}" failed. No retries permitted until 2026-07-30 04:23:24.436384268 +0000 UTC m=+2911.391018927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls") pod "maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6" (UID: "82a63843-3916-43be-a578-6aca95bc91e1") : secret "maas-api-e2e-api-b-2a2d39-serving-cert" not found
Jul 30 04:23:23.936715 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.936686    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a63843-3916-43be-a578-6aca95bc91e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:23.945424 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:23.945402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6bl\" (UniqueName: \"kubernetes.io/projected/82a63843-3916-43be-a578-6aca95bc91e1-kube-api-access-sv6bl\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:24.406678 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.406619    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"]
Jul 30 04:23:24.410776 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.410753    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:24.412745 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.412721    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2a2d39\""
Jul 30 04:23:24.412871 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.412779    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2a2d39-dockercfg-m96g4\""
Jul 30 04:23:24.425852 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.425824    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"]
Jul 30 04:23:24.439734 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.439699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:24.439905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.439789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5wz\" (UniqueName: \"kubernetes.io/projected/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-kube-api-access-8f5wz\") pod \"payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f\" (UID: \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:24.439905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.439828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f\" (UID: \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:24.442563 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.442534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls\") pod \"maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:24.541174 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.541131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5wz\" (UniqueName: \"kubernetes.io/projected/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-kube-api-access-8f5wz\") pod \"payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f\" (UID: \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:24.541390 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.541188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f\" (UID: \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:24.541921 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.541899    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f\" (UID: \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:24.548688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.548663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5wz\" (UniqueName: \"kubernetes.io/projected/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-kube-api-access-8f5wz\") pod \"payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f\" (UID: \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:24.661303 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.661199    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:24.722228 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.722102    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:24.826916 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.826789    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"]
Jul 30 04:23:24.830400 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:23:24.830364    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a63843_3916_43be_a578_6aca95bc91e1.slice/crio-9bb2d489a1d25574c8606dc73a7d8ac5fcbaf15ac92f9489507e82e1289d4727 WatchSource:0}: Error finding container 9bb2d489a1d25574c8606dc73a7d8ac5fcbaf15ac92f9489507e82e1289d4727: Status 404 returned error can't find the container with id 9bb2d489a1d25574c8606dc73a7d8ac5fcbaf15ac92f9489507e82e1289d4727
Jul 30 04:23:24.879780 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:24.879747    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"]
Jul 30 04:23:24.882443 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:23:24.882400    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55d10d1_0e00_4845_86eb_4c6a3fe05fd3.slice/crio-0da4a806a7d823cf2d5ea88f028f7a4d02d52ccc9d999d87a9a90e4ef372c5f5 WatchSource:0}: Error finding container 0da4a806a7d823cf2d5ea88f028f7a4d02d52ccc9d999d87a9a90e4ef372c5f5: Status 404 returned error can't find the container with id 0da4a806a7d823cf2d5ea88f028f7a4d02d52ccc9d999d87a9a90e4ef372c5f5
Jul 30 04:23:25.825507 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:25.825469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6" event={"ID":"82a63843-3916-43be-a578-6aca95bc91e1","Type":"ContainerStarted","Data":"ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f"}
Jul 30 04:23:25.825698 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:25.825531    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6" event={"ID":"82a63843-3916-43be-a578-6aca95bc91e1","Type":"ContainerStarted","Data":"9bb2d489a1d25574c8606dc73a7d8ac5fcbaf15ac92f9489507e82e1289d4727"}
Jul 30 04:23:25.825698 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:25.825564    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:25.827078 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:25.827052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f" event={"ID":"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3","Type":"ContainerStarted","Data":"a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2"}
Jul 30 04:23:25.827468 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:25.827100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f" event={"ID":"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3","Type":"ContainerStarted","Data":"0da4a806a7d823cf2d5ea88f028f7a4d02d52ccc9d999d87a9a90e4ef372c5f5"}
Jul 30 04:23:25.827468 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:25.827140    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:25.840211 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:25.840161    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6" podStartSLOduration=2.415824593 podStartE2EDuration="2.840144917s" totalImagesPullingTime="424.320324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:23:25.839208072 +0000 UTC m=+2912.793842744" watchObservedRunningTime="2026-07-30 04:23:25.840144917 +0000 UTC m=+2912.794779656"
Jul 30 04:23:25.853412 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:25.853356    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f" podStartSLOduration=1.853341056 podStartE2EDuration="1.853341056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:23:25.851104549 +0000 UTC m=+2912.805739221" watchObservedRunningTime="2026-07-30 04:23:25.853341056 +0000 UTC m=+2912.807975788"
Jul 30 04:23:31.838817 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:31.838786    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:23:36.834586 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:36.834535    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f" podUID="d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.71:9004: connect: connection refused"
Jul 30 04:23:46.834547 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:46.834512    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:23:50.342124 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.342079    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"]
Jul 30 04:23:50.345998 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.345974    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"
Jul 30 04:23:50.347838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.347808    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:23:50.355520 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.355486    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"]
Jul 30 04:23:50.501792 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.501751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3292d34-515d-4154-8f73-45c27dd0b36c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks\" (UID: \"d3292d34-515d-4154-8f73-45c27dd0b36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"
Jul 30 04:23:50.603305 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.603187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3292d34-515d-4154-8f73-45c27dd0b36c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks\" (UID: \"d3292d34-515d-4154-8f73-45c27dd0b36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"
Jul 30 04:23:50.603864 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.603835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3292d34-515d-4154-8f73-45c27dd0b36c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks\" (UID: \"d3292d34-515d-4154-8f73-45c27dd0b36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"
Jul 30 04:23:50.657890 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.657846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"
Jul 30 04:23:50.795765 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.795738    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"]
Jul 30 04:23:50.798175 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:23:50.798146    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3292d34_515d_4154_8f73_45c27dd0b36c.slice/crio-060e115629e0daef010cd37bc481f8428f07db15a5ca742c8100b86ee0f8ca4f WatchSource:0}: Error finding container 060e115629e0daef010cd37bc481f8428f07db15a5ca742c8100b86ee0f8ca4f: Status 404 returned error can't find the container with id 060e115629e0daef010cd37bc481f8428f07db15a5ca742c8100b86ee0f8ca4f
Jul 30 04:23:50.964318 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:50.964280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks" event={"ID":"d3292d34-515d-4154-8f73-45c27dd0b36c","Type":"ContainerStarted","Data":"060e115629e0daef010cd37bc481f8428f07db15a5ca742c8100b86ee0f8ca4f"}
Jul 30 04:23:51.972799 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:51.972758    2560 generic.go:356] "Generic (PLEG): container finished" podID="d3292d34-515d-4154-8f73-45c27dd0b36c" containerID="35bed3fca1e3919bff674b4bd9e5158840bbd0c79f74f6a28803266cd0de7ed7" exitCode=0
Jul 30 04:23:51.973221 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:51.972852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks" event={"ID":"d3292d34-515d-4154-8f73-45c27dd0b36c","Type":"ContainerDied","Data":"35bed3fca1e3919bff674b4bd9e5158840bbd0c79f74f6a28803266cd0de7ed7"}
Jul 30 04:23:53.119638 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:53.119612    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"
Jul 30 04:23:53.229771 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:53.229671    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3292d34-515d-4154-8f73-45c27dd0b36c-maas-api-service-ca\") pod \"d3292d34-515d-4154-8f73-45c27dd0b36c\" (UID: \"d3292d34-515d-4154-8f73-45c27dd0b36c\") "
Jul 30 04:23:53.230119 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:53.230085    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3292d34-515d-4154-8f73-45c27dd0b36c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3292d34-515d-4154-8f73-45c27dd0b36c" (UID: "d3292d34-515d-4154-8f73-45c27dd0b36c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:23:53.330459 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:53.330416    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3292d34-515d-4154-8f73-45c27dd0b36c-maas-api-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:23:53.865201 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:53.865110    2560 scope.go:117] "RemoveContainer" containerID="bea8b9ec3ec3428273ed5e929e77120812196622b2b72c29badc236675e521e9"
Jul 30 04:23:53.984665 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:53.984631    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"
Jul 30 04:23:53.984665 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:53.984652    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks" event={"ID":"d3292d34-515d-4154-8f73-45c27dd0b36c","Type":"ContainerDied","Data":"060e115629e0daef010cd37bc481f8428f07db15a5ca742c8100b86ee0f8ca4f"}
Jul 30 04:23:53.984861 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:23:53.984699    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="060e115629e0daef010cd37bc481f8428f07db15a5ca742c8100b86ee0f8ca4f"
Jul 30 04:24:05.764541 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:05.764503    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"]
Jul 30 04:24:05.765037 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:05.764763    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt" podUID="4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" containerName="maas-api" containerID="cri-o://ce3d115dd0001a7148d57d2b010c86971f28945f2be4f5b37d73333881a3eba7" gracePeriod=30
Jul 30 04:24:05.833462 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:05.833424    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"]
Jul 30 04:24:05.833740 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:05.833712    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4" podUID="0a0bc3d3-760f-4eb9-ba10-65bfccd1feba" containerName="payload-processing" containerID="cri-o://29ef8521d8a66277d11f87afef05d9de1c75f2de16359e3e3bf44034bbfd68f0" gracePeriod=30
Jul 30 04:24:05.992404 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:05.992367    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"]
Jul 30 04:24:05.994933 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:05.994903    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c7b98-qfgks"]
Jul 30 04:24:06.052662 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.052635    2560 generic.go:356] "Generic (PLEG): container finished" podID="0a0bc3d3-760f-4eb9-ba10-65bfccd1feba" containerID="29ef8521d8a66277d11f87afef05d9de1c75f2de16359e3e3bf44034bbfd68f0" exitCode=0
Jul 30 04:24:06.052811 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.052713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4" event={"ID":"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba","Type":"ContainerDied","Data":"29ef8521d8a66277d11f87afef05d9de1c75f2de16359e3e3bf44034bbfd68f0"}
Jul 30 04:24:06.054764 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.054743    2560 generic.go:356] "Generic (PLEG): container finished" podID="4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" containerID="ce3d115dd0001a7148d57d2b010c86971f28945f2be4f5b37d73333881a3eba7" exitCode=0
Jul 30 04:24:06.054888 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.054821    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt" event={"ID":"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b","Type":"ContainerDied","Data":"ce3d115dd0001a7148d57d2b010c86971f28945f2be4f5b37d73333881a3eba7"}
Jul 30 04:24:06.098311 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.098286    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:24:06.125448 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.125425    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:24:06.159579 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.159552    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-plugins-config-volume\") pod \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\" (UID: \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\") "
Jul 30 04:24:06.159766 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.159624    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvmb2\" (UniqueName: \"kubernetes.io/projected/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-kube-api-access-zvmb2\") pod \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") "
Jul 30 04:24:06.159766 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.159662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-openshift-service-ca\") pod \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") "
Jul 30 04:24:06.159766 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.159680    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tx8z\" (UniqueName: \"kubernetes.io/projected/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-kube-api-access-8tx8z\") pod \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\" (UID: \"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba\") "
Jul 30 04:24:06.159766 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.159726    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-maas-api-tls\") pod \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\" (UID: \"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b\") "
Jul 30 04:24:06.160014 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.159976    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a0bc3d3-760f-4eb9-ba10-65bfccd1feba" (UID: "0a0bc3d3-760f-4eb9-ba10-65bfccd1feba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:24:06.160078 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.160023    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" (UID: "4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:24:06.160203 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.160185    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:06.160289 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.160204    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:06.161866 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.161834    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-kube-api-access-zvmb2" (OuterVolumeSpecName: "kube-api-access-zvmb2") pod "4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" (UID: "4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b"). InnerVolumeSpecName "kube-api-access-zvmb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:24:06.162395 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.162370    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" (UID: "4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:24:06.162395 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.162378    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-kube-api-access-8tx8z" (OuterVolumeSpecName: "kube-api-access-8tx8z") pod "0a0bc3d3-760f-4eb9-ba10-65bfccd1feba" (UID: "0a0bc3d3-760f-4eb9-ba10-65bfccd1feba"). InnerVolumeSpecName "kube-api-access-8tx8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:24:06.261148 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.261108    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tx8z\" (UniqueName: \"kubernetes.io/projected/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba-kube-api-access-8tx8z\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:06.261148 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.261143    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:06.261395 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:06.261158    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvmb2\" (UniqueName: \"kubernetes.io/projected/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b-kube-api-access-zvmb2\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:07.062416 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.062333    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"
Jul 30 04:24:07.062416 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.062377    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4" event={"ID":"0a0bc3d3-760f-4eb9-ba10-65bfccd1feba","Type":"ContainerDied","Data":"4dd2d5d5883b220038d358514496be5dfbd16f08365074223b425fd3c05c8fc9"}
Jul 30 04:24:07.062911 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.062444    2560 scope.go:117] "RemoveContainer" containerID="29ef8521d8a66277d11f87afef05d9de1c75f2de16359e3e3bf44034bbfd68f0"
Jul 30 04:24:07.063932 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.063843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt" event={"ID":"4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b","Type":"ContainerDied","Data":"92823fef0f23a394ae2087bd30953dde3d8beb4e347e4907d5e7b567c19cd9ad"}
Jul 30 04:24:07.063995 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.063940    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"
Jul 30 04:24:07.073214 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.073165    2560 scope.go:117] "RemoveContainer" containerID="ce3d115dd0001a7148d57d2b010c86971f28945f2be4f5b37d73333881a3eba7"
Jul 30 04:24:07.088946 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.088917    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"]
Jul 30 04:24:07.092306 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.092260    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c7b98-7bc764774d-gt2qt"]
Jul 30 04:24:07.101516 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.101490    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"]
Jul 30 04:24:07.105389 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.105366    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9c7b98-9f5c9794d-vj7z4"]
Jul 30 04:24:07.584929 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.584893    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a0bc3d3-760f-4eb9-ba10-65bfccd1feba" path="/var/lib/kubelet/pods/0a0bc3d3-760f-4eb9-ba10-65bfccd1feba/volumes"
Jul 30 04:24:07.585335 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.585321    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" path="/var/lib/kubelet/pods/4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b/volumes"
Jul 30 04:24:07.585635 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:07.585624    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3292d34-515d-4154-8f73-45c27dd0b36c" path="/var/lib/kubelet/pods/d3292d34-515d-4154-8f73-45c27dd0b36c/volumes"
Jul 30 04:24:12.598446 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598414    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"]
Jul 30 04:24:12.598905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598819    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" containerName="maas-api"
Jul 30 04:24:12.598905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598831    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" containerName="maas-api"
Jul 30 04:24:12.598905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598840    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a0bc3d3-760f-4eb9-ba10-65bfccd1feba" containerName="payload-processing"
Jul 30 04:24:12.598905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598846    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a0bc3d3-760f-4eb9-ba10-65bfccd1feba" containerName="payload-processing"
Jul 30 04:24:12.598905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598860    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3292d34-515d-4154-8f73-45c27dd0b36c" containerName="revoke-keys"
Jul 30 04:24:12.598905 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598866    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3292d34-515d-4154-8f73-45c27dd0b36c" containerName="revoke-keys"
Jul 30 04:24:12.599098 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598927    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3292d34-515d-4154-8f73-45c27dd0b36c" containerName="revoke-keys"
Jul 30 04:24:12.599098 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598935    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dbabc4d-ea3e-4156-b0b9-9fba5aefa88b" containerName="maas-api"
Jul 30 04:24:12.599098 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.598944    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a0bc3d3-760f-4eb9-ba10-65bfccd1feba" containerName="payload-processing"
Jul 30 04:24:12.603697 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.603677    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"
Jul 30 04:24:12.605565 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.605543    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:24:12.609332 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.609270    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"]
Jul 30 04:24:12.722562 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.722520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57740526-9b25-4835-aee1-3ad9e47cf088-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp\" (UID: \"57740526-9b25-4835-aee1-3ad9e47cf088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"
Jul 30 04:24:12.824015 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.823973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57740526-9b25-4835-aee1-3ad9e47cf088-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp\" (UID: \"57740526-9b25-4835-aee1-3ad9e47cf088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"
Jul 30 04:24:12.824687 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.824665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57740526-9b25-4835-aee1-3ad9e47cf088-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp\" (UID: \"57740526-9b25-4835-aee1-3ad9e47cf088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"
Jul 30 04:24:12.916854 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:12.916817    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"
Jul 30 04:24:13.056424 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:13.055364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"]
Jul 30 04:24:13.058877 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:24:13.058838    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57740526_9b25_4835_aee1_3ad9e47cf088.slice/crio-2678c79d31c402d16b146d42a00aca2d0dbf2e96518c9ea0151aaca550990296 WatchSource:0}: Error finding container 2678c79d31c402d16b146d42a00aca2d0dbf2e96518c9ea0151aaca550990296: Status 404 returned error can't find the container with id 2678c79d31c402d16b146d42a00aca2d0dbf2e96518c9ea0151aaca550990296
Jul 30 04:24:13.100902 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:13.100870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp" event={"ID":"57740526-9b25-4835-aee1-3ad9e47cf088","Type":"ContainerStarted","Data":"2678c79d31c402d16b146d42a00aca2d0dbf2e96518c9ea0151aaca550990296"}
Jul 30 04:24:14.111140 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:14.111035    2560 generic.go:356] "Generic (PLEG): container finished" podID="57740526-9b25-4835-aee1-3ad9e47cf088" containerID="bfb85ed061d0c988a84292609100079f9a8caa75ff7625cb2fc394bbb8013399" exitCode=0
Jul 30 04:24:14.111616 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:14.111126    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp" event={"ID":"57740526-9b25-4835-aee1-3ad9e47cf088","Type":"ContainerDied","Data":"bfb85ed061d0c988a84292609100079f9a8caa75ff7625cb2fc394bbb8013399"}
Jul 30 04:24:15.255765 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:15.255739    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"
Jul 30 04:24:15.349470 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:15.349433    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57740526-9b25-4835-aee1-3ad9e47cf088-maas-api-service-ca\") pod \"57740526-9b25-4835-aee1-3ad9e47cf088\" (UID: \"57740526-9b25-4835-aee1-3ad9e47cf088\") "
Jul 30 04:24:15.349824 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:15.349800    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57740526-9b25-4835-aee1-3ad9e47cf088-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57740526-9b25-4835-aee1-3ad9e47cf088" (UID: "57740526-9b25-4835-aee1-3ad9e47cf088"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:24:15.450842 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:15.450805    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57740526-9b25-4835-aee1-3ad9e47cf088-maas-api-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:16.122670 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:16.122630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp" event={"ID":"57740526-9b25-4835-aee1-3ad9e47cf088","Type":"ContainerDied","Data":"2678c79d31c402d16b146d42a00aca2d0dbf2e96518c9ea0151aaca550990296"}
Jul 30 04:24:16.122849 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:16.122691    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2678c79d31c402d16b146d42a00aca2d0dbf2e96518c9ea0151aaca550990296"
Jul 30 04:24:16.122849 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:16.122646    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"
Jul 30 04:24:28.191455 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.191420    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"]
Jul 30 04:24:28.192010 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.191646    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6" podUID="82a63843-3916-43be-a578-6aca95bc91e1" containerName="maas-api" containerID="cri-o://ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f" gracePeriod=30
Jul 30 04:24:28.251490 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.251449    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"]
Jul 30 04:24:28.251765 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.251716    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f" podUID="d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" containerName="payload-pre-processing" containerID="cri-o://a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2" gracePeriod=30
Jul 30 04:24:28.397757 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.397703    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"]
Jul 30 04:24:28.401335 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.401300    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2d39-k2jgp"]
Jul 30 04:24:28.547729 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.547703    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:24:28.554153 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.554129    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:24:28.685701 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.685667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a63843-3916-43be-a578-6aca95bc91e1-openshift-service-ca\") pod \"82a63843-3916-43be-a578-6aca95bc91e1\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") "
Jul 30 04:24:28.685919 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.685724    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-plugins-config-volume\") pod \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\" (UID: \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\") "
Jul 30 04:24:28.685919 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.685778    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls\") pod \"82a63843-3916-43be-a578-6aca95bc91e1\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") "
Jul 30 04:24:28.685919 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.685835    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f5wz\" (UniqueName: \"kubernetes.io/projected/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-kube-api-access-8f5wz\") pod \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\" (UID: \"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3\") "
Jul 30 04:24:28.685919 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.685909    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv6bl\" (UniqueName: \"kubernetes.io/projected/82a63843-3916-43be-a578-6aca95bc91e1-kube-api-access-sv6bl\") pod \"82a63843-3916-43be-a578-6aca95bc91e1\" (UID: \"82a63843-3916-43be-a578-6aca95bc91e1\") "
Jul 30 04:24:28.686292 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.686105    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a63843-3916-43be-a578-6aca95bc91e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82a63843-3916-43be-a578-6aca95bc91e1" (UID: "82a63843-3916-43be-a578-6aca95bc91e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:24:28.686292 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.686215    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" (UID: "d55d10d1-0e00-4845-86eb-4c6a3fe05fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:24:28.686416 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.686304    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a63843-3916-43be-a578-6aca95bc91e1-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:28.688072 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.688043    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82a63843-3916-43be-a578-6aca95bc91e1" (UID: "82a63843-3916-43be-a578-6aca95bc91e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:24:28.688072 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.688048    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-kube-api-access-8f5wz" (OuterVolumeSpecName: "kube-api-access-8f5wz") pod "d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" (UID: "d55d10d1-0e00-4845-86eb-4c6a3fe05fd3"). InnerVolumeSpecName "kube-api-access-8f5wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:24:28.688205 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.688080    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a63843-3916-43be-a578-6aca95bc91e1-kube-api-access-sv6bl" (OuterVolumeSpecName: "kube-api-access-sv6bl") pod "82a63843-3916-43be-a578-6aca95bc91e1" (UID: "82a63843-3916-43be-a578-6aca95bc91e1"). InnerVolumeSpecName "kube-api-access-sv6bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:24:28.787576 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.787488    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:28.787576 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.787521    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a63843-3916-43be-a578-6aca95bc91e1-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:28.787576 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.787533    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f5wz\" (UniqueName: \"kubernetes.io/projected/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3-kube-api-access-8f5wz\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:28.787576 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:28.787543    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6bl\" (UniqueName: \"kubernetes.io/projected/82a63843-3916-43be-a578-6aca95bc91e1-kube-api-access-sv6bl\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:24:29.199382 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.199338    2560 generic.go:356] "Generic (PLEG): container finished" podID="d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" containerID="a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2" exitCode=0
Jul 30 04:24:29.199382 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.199380    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"
Jul 30 04:24:29.199893 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.199430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f" event={"ID":"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3","Type":"ContainerDied","Data":"a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2"}
Jul 30 04:24:29.199893 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.199492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f" event={"ID":"d55d10d1-0e00-4845-86eb-4c6a3fe05fd3","Type":"ContainerDied","Data":"0da4a806a7d823cf2d5ea88f028f7a4d02d52ccc9d999d87a9a90e4ef372c5f5"}
Jul 30 04:24:29.199893 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.199526    2560 scope.go:117] "RemoveContainer" containerID="a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2"
Jul 30 04:24:29.201024 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.200840    2560 generic.go:356] "Generic (PLEG): container finished" podID="82a63843-3916-43be-a578-6aca95bc91e1" containerID="ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f" exitCode=0
Jul 30 04:24:29.201024 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.200876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6" event={"ID":"82a63843-3916-43be-a578-6aca95bc91e1","Type":"ContainerDied","Data":"ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f"}
Jul 30 04:24:29.201024 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.200896    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"
Jul 30 04:24:29.201024 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.200931    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6" event={"ID":"82a63843-3916-43be-a578-6aca95bc91e1","Type":"ContainerDied","Data":"9bb2d489a1d25574c8606dc73a7d8ac5fcbaf15ac92f9489507e82e1289d4727"}
Jul 30 04:24:29.217215 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.217195    2560 scope.go:117] "RemoveContainer" containerID="a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2"
Jul 30 04:24:29.217575 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:24:29.217552    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2\": container with ID starting with a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2 not found: ID does not exist" containerID="a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2"
Jul 30 04:24:29.217633 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.217585    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2"} err="failed to get container status \"a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2\": rpc error: code = NotFound desc = could not find container \"a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2\": container with ID starting with a8feda9bb6818adc8ddb6d877f0c582253d496f433f6ce43ac8545ec0e622bf2 not found: ID does not exist"
Jul 30 04:24:29.217633 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.217603    2560 scope.go:117] "RemoveContainer" containerID="ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f"
Jul 30 04:24:29.226100 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.226075    2560 scope.go:117] "RemoveContainer" containerID="ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f"
Jul 30 04:24:29.226497 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:24:29.226463    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f\": container with ID starting with ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f not found: ID does not exist" containerID="ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f"
Jul 30 04:24:29.226595 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.226501    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f"} err="failed to get container status \"ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f\": rpc error: code = NotFound desc = could not find container \"ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f\": container with ID starting with ae5e0e3ba856ae1bf4c5547ae1394bd2d6407bffdddff4d25fa2f8c9f3d1f27f not found: ID does not exist"
Jul 30 04:24:29.228137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.228117    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"]
Jul 30 04:24:29.231883 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.231863    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a2d39-594ff45664-fzcf6"]
Jul 30 04:24:29.240433 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.240402    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"]
Jul 30 04:24:29.242198 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.242178    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2d39-6db96f59c6-b488f"]
Jul 30 04:24:29.580091 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.580004    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57740526-9b25-4835-aee1-3ad9e47cf088" path="/var/lib/kubelet/pods/57740526-9b25-4835-aee1-3ad9e47cf088/volumes"
Jul 30 04:24:29.580448 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.580433    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a63843-3916-43be-a578-6aca95bc91e1" path="/var/lib/kubelet/pods/82a63843-3916-43be-a578-6aca95bc91e1/volumes"
Jul 30 04:24:29.580756 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:29.580744    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" path="/var/lib/kubelet/pods/d55d10d1-0e00-4845-86eb-4c6a3fe05fd3/volumes"
Jul 30 04:24:40.201221 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.201173    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"]
Jul 30 04:24:40.201840 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.201809    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" containerName="payload-pre-processing"
Jul 30 04:24:40.202007 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.201983    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" containerName="payload-pre-processing"
Jul 30 04:24:40.202095 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.202038    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57740526-9b25-4835-aee1-3ad9e47cf088" containerName="revoke-keys"
Jul 30 04:24:40.202095 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.202051    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57740526-9b25-4835-aee1-3ad9e47cf088" containerName="revoke-keys"
Jul 30 04:24:40.202095 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.202084    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a63843-3916-43be-a578-6aca95bc91e1" containerName="maas-api"
Jul 30 04:24:40.202095 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.202093    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a63843-3916-43be-a578-6aca95bc91e1" containerName="maas-api"
Jul 30 04:24:40.202335 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.202205    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57740526-9b25-4835-aee1-3ad9e47cf088" containerName="revoke-keys"
Jul 30 04:24:40.202335 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.202219    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55d10d1-0e00-4845-86eb-4c6a3fe05fd3" containerName="payload-pre-processing"
Jul 30 04:24:40.202335 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.202247    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a63843-3916-43be-a578-6aca95bc91e1" containerName="maas-api"
Jul 30 04:24:40.205515 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.205493    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.207394 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.207373    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-19ef02-serving-cert\""
Jul 30 04:24:40.217070 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.217047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"]
Jul 30 04:24:40.299921 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.299871    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zq8\" (UniqueName: \"kubernetes.io/projected/21c3bb24-0108-4637-a1f5-40537497c0d0-kube-api-access-q6zq8\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.300082 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.299941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c3bb24-0108-4637-a1f5-40537497c0d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.300082 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.300056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c3bb24-0108-4637-a1f5-40537497c0d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.400644 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.400608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c3bb24-0108-4637-a1f5-40537497c0d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.400796 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.400668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zq8\" (UniqueName: \"kubernetes.io/projected/21c3bb24-0108-4637-a1f5-40537497c0d0-kube-api-access-q6zq8\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.400796 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.400698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c3bb24-0108-4637-a1f5-40537497c0d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.401364 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.401345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c3bb24-0108-4637-a1f5-40537497c0d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.403504 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.403476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c3bb24-0108-4637-a1f5-40537497c0d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.407748 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.407729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zq8\" (UniqueName: \"kubernetes.io/projected/21c3bb24-0108-4637-a1f5-40537497c0d0-kube-api-access-q6zq8\") pod \"maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.518877 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.518774    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:40.664690 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.664649    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"]
Jul 30 04:24:40.667440 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:24:40.667398    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c3bb24_0108_4637_a1f5_40537497c0d0.slice/crio-e77d0d3d1a19d8a0d9c58f0e06d56c43d165b0406815456849a9d500d19c4c8d WatchSource:0}: Error finding container e77d0d3d1a19d8a0d9c58f0e06d56c43d165b0406815456849a9d500d19c4c8d: Status 404 returned error can't find the container with id e77d0d3d1a19d8a0d9c58f0e06d56c43d165b0406815456849a9d500d19c4c8d
Jul 30 04:24:40.901624 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.901582    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"]
Jul 30 04:24:40.906340 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.906312    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:40.909357 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.909328    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-19ef02\""
Jul 30 04:24:40.909504 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.909400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-19ef02-dockercfg-nbw2s\""
Jul 30 04:24:40.922293 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:40.922265    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"]
Jul 30 04:24:41.007358 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.007313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqkm\" (UniqueName: \"kubernetes.io/projected/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-kube-api-access-6zqkm\") pod \"payload-processing-e2e-shared-a-19ef02-679d99786-f75qz\" (UID: \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:41.007587 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.007508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-19ef02-679d99786-f75qz\" (UID: \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:41.108280 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.108229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqkm\" (UniqueName: \"kubernetes.io/projected/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-kube-api-access-6zqkm\") pod \"payload-processing-e2e-shared-a-19ef02-679d99786-f75qz\" (UID: \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:41.108392 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.108343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-19ef02-679d99786-f75qz\" (UID: \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:41.108957 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.108939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-19ef02-679d99786-f75qz\" (UID: \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:41.125409 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.125386    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqkm\" (UniqueName: \"kubernetes.io/projected/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-kube-api-access-6zqkm\") pod \"payload-processing-e2e-shared-a-19ef02-679d99786-f75qz\" (UID: \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:41.217797 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.217716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:41.283342 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.283211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd" event={"ID":"21c3bb24-0108-4637-a1f5-40537497c0d0","Type":"ContainerStarted","Data":"26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b"}
Jul 30 04:24:41.283559 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.283539    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:41.283700 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.283685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd" event={"ID":"21c3bb24-0108-4637-a1f5-40537497c0d0","Type":"ContainerStarted","Data":"e77d0d3d1a19d8a0d9c58f0e06d56c43d165b0406815456849a9d500d19c4c8d"}
Jul 30 04:24:41.301008 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.300958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd" podStartSLOduration=0.885816372 podStartE2EDuration="1.300940784s" totalImagesPullingTime="415.124412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:24:41.298868903 +0000 UTC m=+2988.253503574" watchObservedRunningTime="2026-07-30 04:24:41.300940784 +0000 UTC m=+2988.255575510"
Jul 30 04:24:41.360541 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:41.360479    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"]
Jul 30 04:24:41.363249 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:24:41.363213    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d010dc_3398_4dbe_ae0c_921fff0e5f6e.slice/crio-14f42e5bbfe09c1fb392bcfd334510184e4a2aaf462392b630462c5c3fbfad6b WatchSource:0}: Error finding container 14f42e5bbfe09c1fb392bcfd334510184e4a2aaf462392b630462c5c3fbfad6b: Status 404 returned error can't find the container with id 14f42e5bbfe09c1fb392bcfd334510184e4a2aaf462392b630462c5c3fbfad6b
Jul 30 04:24:42.290331 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:42.290293    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz" event={"ID":"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e","Type":"ContainerStarted","Data":"559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6"}
Jul 30 04:24:42.290804 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:42.290347    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz" event={"ID":"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e","Type":"ContainerStarted","Data":"14f42e5bbfe09c1fb392bcfd334510184e4a2aaf462392b630462c5c3fbfad6b"}
Jul 30 04:24:42.290804 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:42.290572    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:24:42.305850 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:42.305793    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz" podStartSLOduration=2.30577644 podStartE2EDuration="2.30577644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:24:42.305097309 +0000 UTC m=+2989.259732017" watchObservedRunningTime="2026-07-30 04:24:42.30577644 +0000 UTC m=+2989.260411110"
Jul 30 04:24:47.296557 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:47.296529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:24:53.298569 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:24:53.298538    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:25:08.362317 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.362281    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"]
Jul 30 04:25:08.366209 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.366182    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:08.368194 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.368169    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-fcf3e6-serving-cert\""
Jul 30 04:25:08.377682 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.377642    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"]
Jul 30 04:25:08.480504 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.480468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6md\" (UniqueName: \"kubernetes.io/projected/0b1b0b09-a629-4aff-8048-07ff201a3575-kube-api-access-ml6md\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:08.480684 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.480522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:08.480684 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.480616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1b0b09-a629-4aff-8048-07ff201a3575-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:08.581677 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.581634    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6md\" (UniqueName: \"kubernetes.io/projected/0b1b0b09-a629-4aff-8048-07ff201a3575-kube-api-access-ml6md\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:08.581845 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.581688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:08.581845 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.581734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1b0b09-a629-4aff-8048-07ff201a3575-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:08.581918 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:25:08.581884    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-serving-cert: secret "maas-api-e2e-shared-b-fcf3e6-serving-cert" not found
Jul 30 04:25:08.581970 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:25:08.581957    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls podName:0b1b0b09-a629-4aff-8048-07ff201a3575 nodeName:}" failed. No retries permitted until 2026-07-30 04:25:09.081937113 +0000 UTC m=+3016.036571767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls") pod "maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j" (UID: "0b1b0b09-a629-4aff-8048-07ff201a3575") : secret "maas-api-e2e-shared-b-fcf3e6-serving-cert" not found
Jul 30 04:25:08.582530 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.582509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1b0b09-a629-4aff-8048-07ff201a3575-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:08.589991 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:08.589964    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6md\" (UniqueName: \"kubernetes.io/projected/0b1b0b09-a629-4aff-8048-07ff201a3575-kube-api-access-ml6md\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:09.087678 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.087643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:09.090116 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.090093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls\") pod \"maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:09.098855 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.098825    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"]
Jul 30 04:25:09.102699 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.102684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:09.104503 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.104481    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-fcf3e6\""
Jul 30 04:25:09.111193 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.110973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"]
Jul 30 04:25:09.281203 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.281166    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:09.290810 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.290777    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvx65\" (UniqueName: \"kubernetes.io/projected/81c08e06-0896-4db5-a9e7-ea9644e47060-kube-api-access-vvx65\") pod \"payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485\" (UID: \"81c08e06-0896-4db5-a9e7-ea9644e47060\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:09.290955 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.290926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c08e06-0896-4db5-a9e7-ea9644e47060-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485\" (UID: \"81c08e06-0896-4db5-a9e7-ea9644e47060\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:09.391912 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.391871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvx65\" (UniqueName: \"kubernetes.io/projected/81c08e06-0896-4db5-a9e7-ea9644e47060-kube-api-access-vvx65\") pod \"payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485\" (UID: \"81c08e06-0896-4db5-a9e7-ea9644e47060\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:09.392425 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.392013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c08e06-0896-4db5-a9e7-ea9644e47060-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485\" (UID: \"81c08e06-0896-4db5-a9e7-ea9644e47060\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:09.393087 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.393068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c08e06-0896-4db5-a9e7-ea9644e47060-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485\" (UID: \"81c08e06-0896-4db5-a9e7-ea9644e47060\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:09.399917 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.399886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvx65\" (UniqueName: \"kubernetes.io/projected/81c08e06-0896-4db5-a9e7-ea9644e47060-kube-api-access-vvx65\") pod \"payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485\" (UID: \"81c08e06-0896-4db5-a9e7-ea9644e47060\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:09.415874 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.415841    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:09.430931 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.430903    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"]
Jul 30 04:25:09.433143 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:25:09.433106    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1b0b09_a629_4aff_8048_07ff201a3575.slice/crio-09ad2fdda0b049ca9edcf1cbf53ecd5d78b61fa6fbd545a9b45d6710c7788f3f WatchSource:0}: Error finding container 09ad2fdda0b049ca9edcf1cbf53ecd5d78b61fa6fbd545a9b45d6710c7788f3f: Status 404 returned error can't find the container with id 09ad2fdda0b049ca9edcf1cbf53ecd5d78b61fa6fbd545a9b45d6710c7788f3f
Jul 30 04:25:09.440542 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.440515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j" event={"ID":"0b1b0b09-a629-4aff-8048-07ff201a3575","Type":"ContainerStarted","Data":"09ad2fdda0b049ca9edcf1cbf53ecd5d78b61fa6fbd545a9b45d6710c7788f3f"}
Jul 30 04:25:09.768272 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:09.767923    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"]
Jul 30 04:25:10.447135 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:10.447090    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j" event={"ID":"0b1b0b09-a629-4aff-8048-07ff201a3575","Type":"ContainerStarted","Data":"71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd"}
Jul 30 04:25:10.447618 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:10.447213    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:10.448879 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:10.448853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485" event={"ID":"81c08e06-0896-4db5-a9e7-ea9644e47060","Type":"ContainerStarted","Data":"47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931"}
Jul 30 04:25:10.448981 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:10.448903    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485" event={"ID":"81c08e06-0896-4db5-a9e7-ea9644e47060","Type":"ContainerStarted","Data":"1d54fde44feadf8529a7903f872fda403710498b033edea851b2c535c96ce554"}
Jul 30 04:25:10.448981 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:10.448940    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:10.463558 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:10.463513    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j" podStartSLOduration=1.955957856 podStartE2EDuration="2.46350032s" totalImagesPullingTime="507.542464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:25:10.460984178 +0000 UTC m=+3017.415618851" watchObservedRunningTime="2026-07-30 04:25:10.46350032 +0000 UTC m=+3017.418134991"
Jul 30 04:25:10.476442 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:10.476389    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485" podStartSLOduration=1.47637111 podStartE2EDuration="1.47637111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:25:10.473824229 +0000 UTC m=+3017.428458899" watchObservedRunningTime="2026-07-30 04:25:10.47637111 +0000 UTC m=+3017.431005825"
Jul 30 04:25:16.460404 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:16.460365    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:25:21.457435 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:21.457405    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:25:53.911710 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:25:53.911677    2560 scope.go:117] "RemoveContainer" containerID="7e0e5c79aabd8591fa74a3ef5a16ef34cf52f66b4b3e15483ce565b86ff5720f"
Jul 30 04:28:53.947333 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:28:53.947253    2560 scope.go:117] "RemoveContainer" containerID="696ca936b52d29d68bd401d4bc03481d946d0e7d7ecdb11d194363363faaebe4"
Jul 30 04:28:53.956392 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:28:53.956370    2560 scope.go:117] "RemoveContainer" containerID="e10691e8ec307876b729ce614c6b77d400721aa7309a2c0cf13893cdf31fba12"
Jul 30 04:29:22.383034 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.383000    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8rnrb"]
Jul 30 04:29:22.387195 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.387170    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:22.389026 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.389002    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-ghxvb\""
Jul 30 04:29:22.401046 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.401016    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8rnrb"]
Jul 30 04:29:22.523678 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.523637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765md\" (UniqueName: \"kubernetes.io/projected/b6bdbfea-46cd-4e6f-a283-fcc3a83e8626-kube-api-access-765md\") pod \"kuadrant-operator-catalog-8rnrb\" (UID: \"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626\") " pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:22.624599 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.624559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765md\" (UniqueName: \"kubernetes.io/projected/b6bdbfea-46cd-4e6f-a283-fcc3a83e8626-kube-api-access-765md\") pod \"kuadrant-operator-catalog-8rnrb\" (UID: \"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626\") " pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:22.632815 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.632777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-765md\" (UniqueName: \"kubernetes.io/projected/b6bdbfea-46cd-4e6f-a283-fcc3a83e8626-kube-api-access-765md\") pod \"kuadrant-operator-catalog-8rnrb\" (UID: \"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626\") " pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:22.697856 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.697817    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:22.841175 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.841143    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8rnrb"]
Jul 30 04:29:22.843744 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:29:22.843709    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bdbfea_46cd_4e6f_a283_fcc3a83e8626.slice/crio-bb6bf49bd03c30c0e5a44d5d15d203a247d7d8c3b4bcf09cc302c240933fd585 WatchSource:0}: Error finding container bb6bf49bd03c30c0e5a44d5d15d203a247d7d8c3b4bcf09cc302c240933fd585: Status 404 returned error can't find the container with id bb6bf49bd03c30c0e5a44d5d15d203a247d7d8c3b4bcf09cc302c240933fd585
Jul 30 04:29:22.845081 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:22.845064    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:29:23.808526 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:23.808481    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb" event={"ID":"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626","Type":"ContainerStarted","Data":"bb6bf49bd03c30c0e5a44d5d15d203a247d7d8c3b4bcf09cc302c240933fd585"}
Jul 30 04:29:24.815654 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:24.815611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb" event={"ID":"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626","Type":"ContainerStarted","Data":"71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06"}
Jul 30 04:29:24.836356 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:24.836297    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb" podStartSLOduration=1.404845117 podStartE2EDuration="2.836280919s" totalImagesPullingTime="1.431435802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:29:24.833380186 +0000 UTC m=+3271.788014856" watchObservedRunningTime="2026-07-30 04:29:24.836280919 +0000 UTC m=+3271.790915589"
Jul 30 04:29:32.698304 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:32.698186    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:32.698304 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:32.698287    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:32.721697 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:32.721670    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:32.882843 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:32.882812    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:32.945933 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:32.945897    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8rnrb"]
Jul 30 04:29:34.871204 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:34.871130    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb" podUID="b6bdbfea-46cd-4e6f-a283-fcc3a83e8626" containerName="registry-server" containerID="cri-o://71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06" gracePeriod=2
Jul 30 04:29:35.134743 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.134676    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:35.250535 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.250500    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-765md\" (UniqueName: \"kubernetes.io/projected/b6bdbfea-46cd-4e6f-a283-fcc3a83e8626-kube-api-access-765md\") pod \"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626\" (UID: \"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626\") "
Jul 30 04:29:35.252613 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.252580    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bdbfea-46cd-4e6f-a283-fcc3a83e8626-kube-api-access-765md" (OuterVolumeSpecName: "kube-api-access-765md") pod "b6bdbfea-46cd-4e6f-a283-fcc3a83e8626" (UID: "b6bdbfea-46cd-4e6f-a283-fcc3a83e8626"). InnerVolumeSpecName "kube-api-access-765md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:29:35.351727 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.351694    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-765md\" (UniqueName: \"kubernetes.io/projected/b6bdbfea-46cd-4e6f-a283-fcc3a83e8626-kube-api-access-765md\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:29:35.879429 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.879341    2560 generic.go:356] "Generic (PLEG): container finished" podID="b6bdbfea-46cd-4e6f-a283-fcc3a83e8626" containerID="71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06" exitCode=0
Jul 30 04:29:35.879429 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.879396    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb"
Jul 30 04:29:35.879429 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.879420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb" event={"ID":"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626","Type":"ContainerDied","Data":"71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06"}
Jul 30 04:29:35.879888 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.879484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8rnrb" event={"ID":"b6bdbfea-46cd-4e6f-a283-fcc3a83e8626","Type":"ContainerDied","Data":"bb6bf49bd03c30c0e5a44d5d15d203a247d7d8c3b4bcf09cc302c240933fd585"}
Jul 30 04:29:35.879888 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.879509    2560 scope.go:117] "RemoveContainer" containerID="71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06"
Jul 30 04:29:35.889206 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.889184    2560 scope.go:117] "RemoveContainer" containerID="71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06"
Jul 30 04:29:35.889556 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:29:35.889534    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06\": container with ID starting with 71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06 not found: ID does not exist" containerID="71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06"
Jul 30 04:29:35.889629 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.889568    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06"} err="failed to get container status \"71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06\": rpc error: code = NotFound desc = could not find container \"71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06\": container with ID starting with 71ad7ee1cd69f10b27808d531804ebee2831cfa3b04a6e03ac94cf760cce5e06 not found: ID does not exist"
Jul 30 04:29:35.894996 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.894954    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8rnrb"]
Jul 30 04:29:35.898421 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:35.898393    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8rnrb"]
Jul 30 04:29:37.579098 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:37.579063    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6bdbfea-46cd-4e6f-a283-fcc3a83e8626" path="/var/lib/kubelet/pods/b6bdbfea-46cd-4e6f-a283-fcc3a83e8626/volumes"
Jul 30 04:29:53.982300 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:29:53.982265    2560 scope.go:117] "RemoveContainer" containerID="35bed3fca1e3919bff674b4bd9e5158840bbd0c79f74f6a28803266cd0de7ed7"
Jul 30 04:30:00.139162 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.139127    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"]
Jul 30 04:30:00.139878 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.139853    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bdbfea-46cd-4e6f-a283-fcc3a83e8626" containerName="registry-server"
Jul 30 04:30:00.139978 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.139881    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bdbfea-46cd-4e6f-a283-fcc3a83e8626" containerName="registry-server"
Jul 30 04:30:00.140035 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.140018    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bdbfea-46cd-4e6f-a283-fcc3a83e8626" containerName="registry-server"
Jul 30 04:30:00.146671 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.146643    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2"]
Jul 30 04:30:00.146794 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.146760    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"
Jul 30 04:30:00.148800 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.148765    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:30:00.151535 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.151507    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"]
Jul 30 04:30:00.151666 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.151634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2"
Jul 30 04:30:00.162622 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.162600    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"
Jul 30 04:30:00.167010 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.166989    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2"]
Jul 30 04:30:00.167091 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.167031    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"]
Jul 30 04:30:00.167091 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.167061    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"]
Jul 30 04:30:00.167091 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.167084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"
Jul 30 04:30:00.170223 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.170204    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2"
Jul 30 04:30:00.209525 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.209497    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"
Jul 30 04:30:00.334545 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.334025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"]
Jul 30 04:30:00.360407 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.360021    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2"]
Jul 30 04:30:00.362729 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:30:00.362670    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678b6af3_0869_43bf_8b47_5a9784e8c97c.slice/crio-e107e77dc189a550960594c948f18acb455e7b26b5793842dd1ec7f84480126a WatchSource:0}: Error finding container e107e77dc189a550960594c948f18acb455e7b26b5793842dd1ec7f84480126a: Status 404 returned error can't find the container with id e107e77dc189a550960594c948f18acb455e7b26b5793842dd1ec7f84480126a
Jul 30 04:30:00.394059 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:00.394001    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"]
Jul 30 04:30:00.401484 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:30:00.401460    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120b7df7_e763_48a3_b3a7_339c06379452.slice/crio-51e678cedc75a0d8e049e314f4bfa888cedffeb18ff995a91689be987fa06784 WatchSource:0}: Error finding container 51e678cedc75a0d8e049e314f4bfa888cedffeb18ff995a91689be987fa06784: Status 404 returned error can't find the container with id 51e678cedc75a0d8e049e314f4bfa888cedffeb18ff995a91689be987fa06784
Jul 30 04:30:01.020466 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.020354    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2" event={"ID":"678b6af3-0869-43bf-8b47-5a9784e8c97c","Type":"ContainerStarted","Data":"03e2f36499fedb1da8724855bef3b1bd64255ad395357e93b2fcc925221f8cc7"}
Jul 30 04:30:01.020466 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.020424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2" event={"ID":"678b6af3-0869-43bf-8b47-5a9784e8c97c","Type":"ContainerStarted","Data":"e107e77dc189a550960594c948f18acb455e7b26b5793842dd1ec7f84480126a"}
Jul 30 04:30:01.022075 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.022043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p" event={"ID":"120b7df7-e763-48a3-b3a7-339c06379452","Type":"ContainerStarted","Data":"64477f2893da7761e25c20347c5ff0cf33c58b601b41670ae2d45cd6a476d728"}
Jul 30 04:30:01.022215 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.022098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p" event={"ID":"120b7df7-e763-48a3-b3a7-339c06379452","Type":"ContainerStarted","Data":"51e678cedc75a0d8e049e314f4bfa888cedffeb18ff995a91689be987fa06784"}
Jul 30 04:30:01.023914 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.023883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz" event={"ID":"0da710bc-d033-4bc4-96f0-7363e9dca177","Type":"ContainerStarted","Data":"ac0ae7e2b27b1a8f3dfbb5668b9a42fbb856a687a85d4368f4d545c3fc809d05"}
Jul 30 04:30:01.024068 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.023936    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz" event={"ID":"0da710bc-d033-4bc4-96f0-7363e9dca177","Type":"ContainerStarted","Data":"67f3e041254b966c6cc51b206f7b01179507d85bf36022c6793463b42bf97aa4"}
Jul 30 04:30:01.035677 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.035629    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2" podStartSLOduration=1.035616115 podStartE2EDuration="1.035616115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:01.033121053 +0000 UTC m=+3307.987755736" watchObservedRunningTime="2026-07-30 04:30:01.035616115 +0000 UTC m=+3307.990250786"
Jul 30 04:30:01.047339 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.047279    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz" podStartSLOduration=1.047261229 podStartE2EDuration="1.047261229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:01.045391055 +0000 UTC m=+3308.000025726" watchObservedRunningTime="2026-07-30 04:30:01.047261229 +0000 UTC m=+3308.001895899"
Jul 30 04:30:01.058688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:01.058642    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p" podStartSLOduration=1.058628271 podStartE2EDuration="1.058628271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:01.057212595 +0000 UTC m=+3308.011847266" watchObservedRunningTime="2026-07-30 04:30:01.058628271 +0000 UTC m=+3308.013262942"
Jul 30 04:30:02.031033 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:02.030998    2560 generic.go:356] "Generic (PLEG): container finished" podID="678b6af3-0869-43bf-8b47-5a9784e8c97c" containerID="03e2f36499fedb1da8724855bef3b1bd64255ad395357e93b2fcc925221f8cc7" exitCode=0
Jul 30 04:30:02.031561 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:02.031101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2" event={"ID":"678b6af3-0869-43bf-8b47-5a9784e8c97c","Type":"ContainerDied","Data":"03e2f36499fedb1da8724855bef3b1bd64255ad395357e93b2fcc925221f8cc7"}
Jul 30 04:30:02.032829 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:02.032802    2560 generic.go:356] "Generic (PLEG): container finished" podID="120b7df7-e763-48a3-b3a7-339c06379452" containerID="64477f2893da7761e25c20347c5ff0cf33c58b601b41670ae2d45cd6a476d728" exitCode=0
Jul 30 04:30:02.032943 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:02.032883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p" event={"ID":"120b7df7-e763-48a3-b3a7-339c06379452","Type":"ContainerDied","Data":"64477f2893da7761e25c20347c5ff0cf33c58b601b41670ae2d45cd6a476d728"}
Jul 30 04:30:02.034431 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:02.034394    2560 generic.go:356] "Generic (PLEG): container finished" podID="0da710bc-d033-4bc4-96f0-7363e9dca177" containerID="ac0ae7e2b27b1a8f3dfbb5668b9a42fbb856a687a85d4368f4d545c3fc809d05" exitCode=0
Jul 30 04:30:02.034524 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:02.034450    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz" event={"ID":"0da710bc-d033-4bc4-96f0-7363e9dca177","Type":"ContainerDied","Data":"ac0ae7e2b27b1a8f3dfbb5668b9a42fbb856a687a85d4368f4d545c3fc809d05"}
Jul 30 04:30:03.204617 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:03.204585    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2"
Jul 30 04:30:03.235498 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:03.235471    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"
Jul 30 04:30:03.251110 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:03.251078    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"
Jul 30 04:30:04.046546 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.046516    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"
Jul 30 04:30:04.047008 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.046515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p" event={"ID":"120b7df7-e763-48a3-b3a7-339c06379452","Type":"ContainerDied","Data":"51e678cedc75a0d8e049e314f4bfa888cedffeb18ff995a91689be987fa06784"}
Jul 30 04:30:04.047008 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.046664    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e678cedc75a0d8e049e314f4bfa888cedffeb18ff995a91689be987fa06784"
Jul 30 04:30:04.047864 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.047844    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"
Jul 30 04:30:04.047975 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.047893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz" event={"ID":"0da710bc-d033-4bc4-96f0-7363e9dca177","Type":"ContainerDied","Data":"67f3e041254b966c6cc51b206f7b01179507d85bf36022c6793463b42bf97aa4"}
Jul 30 04:30:04.047975 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.047939    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f3e041254b966c6cc51b206f7b01179507d85bf36022c6793463b42bf97aa4"
Jul 30 04:30:04.049465 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.049447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2" event={"ID":"678b6af3-0869-43bf-8b47-5a9784e8c97c","Type":"ContainerDied","Data":"e107e77dc189a550960594c948f18acb455e7b26b5793842dd1ec7f84480126a"}
Jul 30 04:30:04.049566 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.049481    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e107e77dc189a550960594c948f18acb455e7b26b5793842dd1ec7f84480126a"
Jul 30 04:30:04.049566 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:04.049492    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-zk6b2"
Jul 30 04:30:54.009363 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:30:54.009330    2560 scope.go:117] "RemoveContainer" containerID="bfb85ed061d0c988a84292609100079f9a8caa75ff7625cb2fc394bbb8013399"
Jul 30 04:31:31.243354 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243316    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"]
Jul 30 04:31:31.243846 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243725    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="120b7df7-e763-48a3-b3a7-339c06379452" containerName="cleanup"
Jul 30 04:31:31.243846 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243735    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="120b7df7-e763-48a3-b3a7-339c06379452" containerName="cleanup"
Jul 30 04:31:31.243846 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243751    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da710bc-d033-4bc4-96f0-7363e9dca177" containerName="cleanup"
Jul 30 04:31:31.243846 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243756    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da710bc-d033-4bc4-96f0-7363e9dca177" containerName="cleanup"
Jul 30 04:31:31.243846 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243770    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678b6af3-0869-43bf-8b47-5a9784e8c97c" containerName="cleanup"
Jul 30 04:31:31.243846 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243775    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="678b6af3-0869-43bf-8b47-5a9784e8c97c" containerName="cleanup"
Jul 30 04:31:31.243846 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243837    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="678b6af3-0869-43bf-8b47-5a9784e8c97c" containerName="cleanup"
Jul 30 04:31:31.243846 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243845    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="120b7df7-e763-48a3-b3a7-339c06379452" containerName="cleanup"
Jul 30 04:31:31.244116 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.243854    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da710bc-d033-4bc4-96f0-7363e9dca177" containerName="cleanup"
Jul 30 04:31:31.247184 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.247166    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.249150 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.249128    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-19ef02\"/\"kube-root-ca.crt\""
Jul 30 04:31:31.249409 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.249389    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-19ef02\"/\"default-dockercfg-5wkr8\""
Jul 30 04:31:31.249544 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.249524    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-19ef02\"/\"sub-test-model-19ef02-kserve-self-signed-certs\""
Jul 30 04:31:31.249631 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.249584    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-19ef02\"/\"openshift-service-ca.crt\""
Jul 30 04:31:31.255629 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.255600    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"]
Jul 30 04:31:31.384932 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.384893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqn2\" (UniqueName: \"kubernetes.io/projected/99872576-9bec-4842-ba74-30da303e723b-kube-api-access-vmqn2\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.384932 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.384936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99872576-9bec-4842-ba74-30da303e723b-tls-certs\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.385179 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.384962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-model-cache\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.385179 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.385072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-dshm\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.385179 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.385106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-home\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.385179 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.385169    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-tmp-dir\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.485937 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.485898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-dshm\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.485937 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.485944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-home\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.486193 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.485986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-tmp-dir\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.486193 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.486079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqn2\" (UniqueName: \"kubernetes.io/projected/99872576-9bec-4842-ba74-30da303e723b-kube-api-access-vmqn2\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.486193 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.486110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99872576-9bec-4842-ba74-30da303e723b-tls-certs\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.486193 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.486140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-model-cache\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.486474 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.486447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-home\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.486538 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.486474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-tmp-dir\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.486538 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.486529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-model-cache\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.488284 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.488252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-dshm\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.488532 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.488513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99872576-9bec-4842-ba74-30da303e723b-tls-certs\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.492985 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.492963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqn2\" (UniqueName: \"kubernetes.io/projected/99872576-9bec-4842-ba74-30da303e723b-kube-api-access-vmqn2\") pod \"sub-test-model-19ef02-kserve-58988bc685-zsvdk\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") " pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.557460 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.557360    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:31.697248 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:31.697186    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"]
Jul 30 04:31:31.701219 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:31:31.701184    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99872576_9bec_4842_ba74_30da303e723b.slice/crio-7edf63cfd5d8b41db2abe2870af415b717c6b11329616d83975a36e285ab1dc2 WatchSource:0}: Error finding container 7edf63cfd5d8b41db2abe2870af415b717c6b11329616d83975a36e285ab1dc2: Status 404 returned error can't find the container with id 7edf63cfd5d8b41db2abe2870af415b717c6b11329616d83975a36e285ab1dc2
Jul 30 04:31:32.518855 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:32.518812    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk" event={"ID":"99872576-9bec-4842-ba74-30da303e723b","Type":"ContainerStarted","Data":"b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5"}
Jul 30 04:31:32.518855 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:32.518857    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk" event={"ID":"99872576-9bec-4842-ba74-30da303e723b","Type":"ContainerStarted","Data":"7edf63cfd5d8b41db2abe2870af415b717c6b11329616d83975a36e285ab1dc2"}
Jul 30 04:31:32.534651 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:32.534600    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk" podStartSLOduration=1.53458501 podStartE2EDuration="1.53458501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:31:32.532821536 +0000 UTC m=+3399.487456205" watchObservedRunningTime="2026-07-30 04:31:32.53458501 +0000 UTC m=+3399.489219679"
Jul 30 04:31:41.557611 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:41.557571    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:41.557980 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:41.557871    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:41.579051 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:41.579022    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:31:42.581092 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:31:42.581058    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:33:38.386106 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:38.386006    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"]
Jul 30 04:33:53.405940 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.405897    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk" podUID="99872576-9bec-4842-ba74-30da303e723b" containerName="main" containerID="cri-o://b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5" gracePeriod=45
Jul 30 04:33:53.664480 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.664401    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:33:53.814586 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.814549    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-tmp-dir\") pod \"99872576-9bec-4842-ba74-30da303e723b\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") "
Jul 30 04:33:53.814586 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.814601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-dshm\") pod \"99872576-9bec-4842-ba74-30da303e723b\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") "
Jul 30 04:33:53.814841 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.814640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-home\") pod \"99872576-9bec-4842-ba74-30da303e723b\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") "
Jul 30 04:33:53.814841 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.814676    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmqn2\" (UniqueName: \"kubernetes.io/projected/99872576-9bec-4842-ba74-30da303e723b-kube-api-access-vmqn2\") pod \"99872576-9bec-4842-ba74-30da303e723b\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") "
Jul 30 04:33:53.814841 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.814706    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99872576-9bec-4842-ba74-30da303e723b-tls-certs\") pod \"99872576-9bec-4842-ba74-30da303e723b\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") "
Jul 30 04:33:53.814993 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.814857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-model-cache\") pod \"99872576-9bec-4842-ba74-30da303e723b\" (UID: \"99872576-9bec-4842-ba74-30da303e723b\") "
Jul 30 04:33:53.814993 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.814884    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-home" (OuterVolumeSpecName: "home") pod "99872576-9bec-4842-ba74-30da303e723b" (UID: "99872576-9bec-4842-ba74-30da303e723b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:33:53.814993 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.814899    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99872576-9bec-4842-ba74-30da303e723b" (UID: "99872576-9bec-4842-ba74-30da303e723b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:33:53.815158 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.815129    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-model-cache" (OuterVolumeSpecName: "model-cache") pod "99872576-9bec-4842-ba74-30da303e723b" (UID: "99872576-9bec-4842-ba74-30da303e723b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:33:53.815311 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.815296    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-model-cache\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:33:53.815370 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.815340    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-tmp-dir\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:33:53.815370 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.815354    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-home\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:33:53.817141 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.817119    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-dshm" (OuterVolumeSpecName: "dshm") pod "99872576-9bec-4842-ba74-30da303e723b" (UID: "99872576-9bec-4842-ba74-30da303e723b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:33:53.817403 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.817383    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99872576-9bec-4842-ba74-30da303e723b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99872576-9bec-4842-ba74-30da303e723b" (UID: "99872576-9bec-4842-ba74-30da303e723b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:33:53.817470 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.817384    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99872576-9bec-4842-ba74-30da303e723b-kube-api-access-vmqn2" (OuterVolumeSpecName: "kube-api-access-vmqn2") pod "99872576-9bec-4842-ba74-30da303e723b" (UID: "99872576-9bec-4842-ba74-30da303e723b"). InnerVolumeSpecName "kube-api-access-vmqn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:33:53.915993 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.915893    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99872576-9bec-4842-ba74-30da303e723b-dshm\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:33:53.915993 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.915927    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmqn2\" (UniqueName: \"kubernetes.io/projected/99872576-9bec-4842-ba74-30da303e723b-kube-api-access-vmqn2\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:33:53.915993 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:53.915938    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99872576-9bec-4842-ba74-30da303e723b-tls-certs\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:33:54.291679 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.291590    2560 generic.go:356] "Generic (PLEG): container finished" podID="99872576-9bec-4842-ba74-30da303e723b" containerID="b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5" exitCode=0
Jul 30 04:33:54.291679 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.291639    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"
Jul 30 04:33:54.291882 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.291684    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk" event={"ID":"99872576-9bec-4842-ba74-30da303e723b","Type":"ContainerDied","Data":"b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5"}
Jul 30 04:33:54.291882 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.291728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk" event={"ID":"99872576-9bec-4842-ba74-30da303e723b","Type":"ContainerDied","Data":"7edf63cfd5d8b41db2abe2870af415b717c6b11329616d83975a36e285ab1dc2"}
Jul 30 04:33:54.291882 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.291754    2560 scope.go:117] "RemoveContainer" containerID="b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5"
Jul 30 04:33:54.301506 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.301481    2560 scope.go:117] "RemoveContainer" containerID="b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5"
Jul 30 04:33:54.301784 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:33:54.301763    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5\": container with ID starting with b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5 not found: ID does not exist" containerID="b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5"
Jul 30 04:33:54.301831 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.301793    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5"} err="failed to get container status \"b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5\": rpc error: code = NotFound desc = could not find container \"b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5\": container with ID starting with b6f13e788cf02d9bf2031207236852402c631f9361e08063f3e5456383fa19a5 not found: ID does not exist"
Jul 30 04:33:54.313400 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.313370    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"]
Jul 30 04:33:54.317297 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:54.317263    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-19ef02/sub-test-model-19ef02-kserve-58988bc685-zsvdk"]
Jul 30 04:33:55.580671 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:33:55.580633    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99872576-9bec-4842-ba74-30da303e723b" path="/var/lib/kubelet/pods/99872576-9bec-4842-ba74-30da303e723b/volumes"
Jul 30 04:36:35.078646 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.078602    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"]
Jul 30 04:36:35.079370 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.079349    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99872576-9bec-4842-ba74-30da303e723b" containerName="main"
Jul 30 04:36:35.079464 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.079373    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="99872576-9bec-4842-ba74-30da303e723b" containerName="main"
Jul 30 04:36:35.079524 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.079504    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="99872576-9bec-4842-ba74-30da303e723b" containerName="main"
Jul 30 04:36:35.083455 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.083429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.085469 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.085419    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-77b2c7-serving-cert\""
Jul 30 04:36:35.093109 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.093078    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"]
Jul 30 04:36:35.110383 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.110350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvslm\" (UniqueName: \"kubernetes.io/projected/e49bc11f-ddcd-4d12-9900-2456c87de2ac-kube-api-access-gvslm\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.110560 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.110415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.110560 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.110457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49bc11f-ddcd-4d12-9900-2456c87de2ac-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.211847 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.211798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.212064 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.211891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49bc11f-ddcd-4d12-9900-2456c87de2ac-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.212064 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:36:35.211941    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-serving-cert: secret "maas-api-e2e-ipp-a-77b2c7-serving-cert" not found
Jul 30 04:36:35.212064 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.211985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvslm\" (UniqueName: \"kubernetes.io/projected/e49bc11f-ddcd-4d12-9900-2456c87de2ac-kube-api-access-gvslm\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.212064 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:36:35.212032    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls podName:e49bc11f-ddcd-4d12-9900-2456c87de2ac nodeName:}" failed. No retries permitted until 2026-07-30 04:36:35.712012671 +0000 UTC m=+3702.666647320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls") pod "maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl" (UID: "e49bc11f-ddcd-4d12-9900-2456c87de2ac") : secret "maas-api-e2e-ipp-a-77b2c7-serving-cert" not found
Jul 30 04:36:35.212592 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.212571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49bc11f-ddcd-4d12-9900-2456c87de2ac-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.223482 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.223447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvslm\" (UniqueName: \"kubernetes.io/projected/e49bc11f-ddcd-4d12-9900-2456c87de2ac-kube-api-access-gvslm\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.716024 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.715986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.718561 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.718460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:35.820658 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.820625    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"]
Jul 30 04:36:35.824592 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.824570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:35.826327 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.826301    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-77b2c7\""
Jul 30 04:36:35.826327 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.826322    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-77b2c7-dockercfg-k48x7\""
Jul 30 04:36:35.836661 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.836632    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"]
Jul 30 04:36:35.844608 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.844584    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"]
Jul 30 04:36:35.848272 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.848225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:35.858437 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.858405    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"]
Jul 30 04:36:35.917531 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.917494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szk7p\" (UniqueName: \"kubernetes.io/projected/20f2e95e-420a-46e3-b833-3944d81f0214-kube-api-access-szk7p\") pod \"payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m\" (UID: \"20f2e95e-420a-46e3-b833-3944d81f0214\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:35.917690 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.917535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f2e95e-420a-46e3-b833-3944d81f0214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m\" (UID: \"20f2e95e-420a-46e3-b833-3944d81f0214\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:35.917690 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.917572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bqn\" (UniqueName: \"kubernetes.io/projected/ba26bdfb-3f19-4dd1-a037-05518057fc2d-kube-api-access-l2bqn\") pod \"payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp\" (UID: \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:35.917690 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:35.917595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba26bdfb-3f19-4dd1-a037-05518057fc2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp\" (UID: \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:36.001117 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.001012    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:36.019026 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.018986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szk7p\" (UniqueName: \"kubernetes.io/projected/20f2e95e-420a-46e3-b833-3944d81f0214-kube-api-access-szk7p\") pod \"payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m\" (UID: \"20f2e95e-420a-46e3-b833-3944d81f0214\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:36.019162 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.019035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f2e95e-420a-46e3-b833-3944d81f0214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m\" (UID: \"20f2e95e-420a-46e3-b833-3944d81f0214\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:36.019162 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.019071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bqn\" (UniqueName: \"kubernetes.io/projected/ba26bdfb-3f19-4dd1-a037-05518057fc2d-kube-api-access-l2bqn\") pod \"payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp\" (UID: \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:36.019162 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.019099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba26bdfb-3f19-4dd1-a037-05518057fc2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp\" (UID: \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:36.019952 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.019927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f2e95e-420a-46e3-b833-3944d81f0214-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m\" (UID: \"20f2e95e-420a-46e3-b833-3944d81f0214\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:36.020080 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.019977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba26bdfb-3f19-4dd1-a037-05518057fc2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp\" (UID: \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:36.026889 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.026865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szk7p\" (UniqueName: \"kubernetes.io/projected/20f2e95e-420a-46e3-b833-3944d81f0214-kube-api-access-szk7p\") pod \"payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m\" (UID: \"20f2e95e-420a-46e3-b833-3944d81f0214\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:36.026997 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.026868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bqn\" (UniqueName: \"kubernetes.io/projected/ba26bdfb-3f19-4dd1-a037-05518057fc2d-kube-api-access-l2bqn\") pod \"payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp\" (UID: \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:36.137094 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.137020    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:36.158104 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.154608    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"]
Jul 30 04:36:36.159809 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.159781    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:36.160194 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:36:36.160169    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49bc11f_ddcd_4d12_9900_2456c87de2ac.slice/crio-bb9da3361064d61885feecf0652c54320def704ee5e500444724d1b2e18ef4da WatchSource:0}: Error finding container bb9da3361064d61885feecf0652c54320def704ee5e500444724d1b2e18ef4da: Status 404 returned error can't find the container with id bb9da3361064d61885feecf0652c54320def704ee5e500444724d1b2e18ef4da
Jul 30 04:36:36.161640 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.161619    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:36:36.174423 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.174332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl" event={"ID":"e49bc11f-ddcd-4d12-9900-2456c87de2ac","Type":"ContainerStarted","Data":"bb9da3361064d61885feecf0652c54320def704ee5e500444724d1b2e18ef4da"}
Jul 30 04:36:36.306673 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.306640    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"]
Jul 30 04:36:36.310436 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:36:36.310408    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f2e95e_420a_46e3_b833_3944d81f0214.slice/crio-6d1e57408f5ce1b58a56c2e1ecad68d9ee7d256a5cc9ca60bcb24f09f978fee9 WatchSource:0}: Error finding container 6d1e57408f5ce1b58a56c2e1ecad68d9ee7d256a5cc9ca60bcb24f09f978fee9: Status 404 returned error can't find the container with id 6d1e57408f5ce1b58a56c2e1ecad68d9ee7d256a5cc9ca60bcb24f09f978fee9
Jul 30 04:36:36.336101 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:36.336070    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"]
Jul 30 04:36:36.339782 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:36:36.339754    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba26bdfb_3f19_4dd1_a037_05518057fc2d.slice/crio-40a8a7feb14f77740dc54bf149900205d81f0a7fa66183a3b9e19fe476fb44be WatchSource:0}: Error finding container 40a8a7feb14f77740dc54bf149900205d81f0a7fa66183a3b9e19fe476fb44be: Status 404 returned error can't find the container with id 40a8a7feb14f77740dc54bf149900205d81f0a7fa66183a3b9e19fe476fb44be
Jul 30 04:36:37.187678 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.187638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl" event={"ID":"e49bc11f-ddcd-4d12-9900-2456c87de2ac","Type":"ContainerStarted","Data":"1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e"}
Jul 30 04:36:37.188182 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.188005    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:37.189306 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.189275    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m" event={"ID":"20f2e95e-420a-46e3-b833-3944d81f0214","Type":"ContainerStarted","Data":"427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66"}
Jul 30 04:36:37.189448 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.189333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m" event={"ID":"20f2e95e-420a-46e3-b833-3944d81f0214","Type":"ContainerStarted","Data":"6d1e57408f5ce1b58a56c2e1ecad68d9ee7d256a5cc9ca60bcb24f09f978fee9"}
Jul 30 04:36:37.189448 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.189407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:37.190969 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.190947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp" event={"ID":"ba26bdfb-3f19-4dd1-a037-05518057fc2d","Type":"ContainerStarted","Data":"31d3f4ced4745c594afd5f834d735258f46614b7ee0d7aa3e49a87a6a30f9f45"}
Jul 30 04:36:37.191117 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.190982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp" event={"ID":"ba26bdfb-3f19-4dd1-a037-05518057fc2d","Type":"ContainerStarted","Data":"40a8a7feb14f77740dc54bf149900205d81f0a7fa66183a3b9e19fe476fb44be"}
Jul 30 04:36:37.191117 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.191095    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:36:37.206641 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.206594    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl" podStartSLOduration=1.785242238 podStartE2EDuration="2.206579131s" totalImagesPullingTime="421.336893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:36:37.202141429 +0000 UTC m=+3704.156776100" watchObservedRunningTime="2026-07-30 04:36:37.206579131 +0000 UTC m=+3704.161213801"
Jul 30 04:36:37.217501 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.217411    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp" podStartSLOduration=2.217391047 podStartE2EDuration="2.217391047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:36:37.216604775 +0000 UTC m=+3704.171239447" watchObservedRunningTime="2026-07-30 04:36:37.217391047 +0000 UTC m=+3704.172025719"
Jul 30 04:36:37.233174 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:37.233125    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m" podStartSLOduration=2.233108543 podStartE2EDuration="2.233108543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:36:37.229666577 +0000 UTC m=+3704.184301249" watchObservedRunningTime="2026-07-30 04:36:37.233108543 +0000 UTC m=+3704.187743214"
Jul 30 04:36:43.202650 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:43.202616    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:36:48.199197 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:48.199166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:36:48.199619 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:36:48.199301    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:37:04.316027 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.315992    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"]
Jul 30 04:37:04.321093 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.321068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.323004 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.322868    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1d8c4c-serving-cert\""
Jul 30 04:37:04.330278 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.329928    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"]
Jul 30 04:37:04.408539 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.408505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/badb439f-3412-43a3-9285-25b112eb6f54-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.408744 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.408554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn99x\" (UniqueName: \"kubernetes.io/projected/badb439f-3412-43a3-9285-25b112eb6f54-kube-api-access-pn99x\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.408744 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.408653    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.509775 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.509735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/badb439f-3412-43a3-9285-25b112eb6f54-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.509775 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.509783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn99x\" (UniqueName: \"kubernetes.io/projected/badb439f-3412-43a3-9285-25b112eb6f54-kube-api-access-pn99x\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.509994 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.509842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.510032 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:37:04.509999    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-serving-cert: secret "maas-api-e2e-ipp-b-1d8c4c-serving-cert" not found
Jul 30 04:37:04.510085 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:37:04.510073    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls podName:badb439f-3412-43a3-9285-25b112eb6f54 nodeName:}" failed. No retries permitted until 2026-07-30 04:37:05.010053575 +0000 UTC m=+3731.964688244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls") pod "maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc" (UID: "badb439f-3412-43a3-9285-25b112eb6f54") : secret "maas-api-e2e-ipp-b-1d8c4c-serving-cert" not found
Jul 30 04:37:04.510583 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.510557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/badb439f-3412-43a3-9285-25b112eb6f54-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.519940 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.519906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn99x\" (UniqueName: \"kubernetes.io/projected/badb439f-3412-43a3-9285-25b112eb6f54-kube-api-access-pn99x\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:04.912059 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.912020    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"]
Jul 30 04:37:04.916103 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.916067    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:04.918472 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.918447    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1d8c4c-dockercfg-2c55t\""
Jul 30 04:37:04.918472 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.918467    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1d8c4c\""
Jul 30 04:37:04.925082 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.924817    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"]
Jul 30 04:37:04.950884 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.950847    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"]
Jul 30 04:37:04.954463 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.954442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:04.967042 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:04.967014    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"]
Jul 30 04:37:05.015002 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.014967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:05.015180 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.015012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgz2p\" (UniqueName: \"kubernetes.io/projected/33b90a06-0cf1-4193-9eac-760c4742b229-kube-api-access-tgz2p\") pod \"payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k\" (UID: \"33b90a06-0cf1-4193-9eac-760c4742b229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:05.015180 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.015116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b90a06-0cf1-4193-9eac-760c4742b229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k\" (UID: \"33b90a06-0cf1-4193-9eac-760c4742b229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:05.017562 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.017533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:05.116711 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.116673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgz2p\" (UniqueName: \"kubernetes.io/projected/33b90a06-0cf1-4193-9eac-760c4742b229-kube-api-access-tgz2p\") pod \"payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k\" (UID: \"33b90a06-0cf1-4193-9eac-760c4742b229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:05.116884 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.116834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b90a06-0cf1-4193-9eac-760c4742b229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k\" (UID: \"33b90a06-0cf1-4193-9eac-760c4742b229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:05.116884 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.116870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70499c2-2753-4155-ad97-504cff67fb8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx\" (UID: \"d70499c2-2753-4155-ad97-504cff67fb8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:05.116997 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.116926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2grm\" (UniqueName: \"kubernetes.io/projected/d70499c2-2753-4155-ad97-504cff67fb8a-kube-api-access-z2grm\") pod \"payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx\" (UID: \"d70499c2-2753-4155-ad97-504cff67fb8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:05.117663 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.117642    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b90a06-0cf1-4193-9eac-760c4742b229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k\" (UID: \"33b90a06-0cf1-4193-9eac-760c4742b229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:05.128639 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.128609    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgz2p\" (UniqueName: \"kubernetes.io/projected/33b90a06-0cf1-4193-9eac-760c4742b229-kube-api-access-tgz2p\") pod \"payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k\" (UID: \"33b90a06-0cf1-4193-9eac-760c4742b229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:05.217763 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.217666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2grm\" (UniqueName: \"kubernetes.io/projected/d70499c2-2753-4155-ad97-504cff67fb8a-kube-api-access-z2grm\") pod \"payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx\" (UID: \"d70499c2-2753-4155-ad97-504cff67fb8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:05.217936 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.217773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70499c2-2753-4155-ad97-504cff67fb8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx\" (UID: \"d70499c2-2753-4155-ad97-504cff67fb8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:05.218519 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.218499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70499c2-2753-4155-ad97-504cff67fb8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx\" (UID: \"d70499c2-2753-4155-ad97-504cff67fb8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:05.224839 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.224817    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2grm\" (UniqueName: \"kubernetes.io/projected/d70499c2-2753-4155-ad97-504cff67fb8a-kube-api-access-z2grm\") pod \"payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx\" (UID: \"d70499c2-2753-4155-ad97-504cff67fb8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:05.229684 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.229657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:05.244799 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.244772    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:05.265977 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.265934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:05.408314 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.408284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"]
Jul 30 04:37:05.411408 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:37:05.411373    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b90a06_0cf1_4193_9eac_760c4742b229.slice/crio-e804697abd259fd5c777290883926ccde6cf924a7400b5692cf59f5790b01274 WatchSource:0}: Error finding container e804697abd259fd5c777290883926ccde6cf924a7400b5692cf59f5790b01274: Status 404 returned error can't find the container with id e804697abd259fd5c777290883926ccde6cf924a7400b5692cf59f5790b01274
Jul 30 04:37:05.445187 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.445137    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"]
Jul 30 04:37:05.449898 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:37:05.449855    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadb439f_3412_43a3_9285_25b112eb6f54.slice/crio-44b04d3c06c9338c8f861debc91638958dbd97b265ea0917a1676c5e56a9e29f WatchSource:0}: Error finding container 44b04d3c06c9338c8f861debc91638958dbd97b265ea0917a1676c5e56a9e29f: Status 404 returned error can't find the container with id 44b04d3c06c9338c8f861debc91638958dbd97b265ea0917a1676c5e56a9e29f
Jul 30 04:37:05.679695 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:05.679663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"]
Jul 30 04:37:05.681200 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:37:05.681170    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70499c2_2753_4155_ad97_504cff67fb8a.slice/crio-7225e72ae35fbd0fe5f5e93d2a88f09f3943c3f91385ef085888010d1b3c1a08 WatchSource:0}: Error finding container 7225e72ae35fbd0fe5f5e93d2a88f09f3943c3f91385ef085888010d1b3c1a08: Status 404 returned error can't find the container with id 7225e72ae35fbd0fe5f5e93d2a88f09f3943c3f91385ef085888010d1b3c1a08
Jul 30 04:37:06.365742 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.365709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx" event={"ID":"d70499c2-2753-4155-ad97-504cff67fb8a","Type":"ContainerStarted","Data":"a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b"}
Jul 30 04:37:06.365912 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.365766    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx" event={"ID":"d70499c2-2753-4155-ad97-504cff67fb8a","Type":"ContainerStarted","Data":"7225e72ae35fbd0fe5f5e93d2a88f09f3943c3f91385ef085888010d1b3c1a08"}
Jul 30 04:37:06.365912 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.365791    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:06.367355 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.367321    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k" event={"ID":"33b90a06-0cf1-4193-9eac-760c4742b229","Type":"ContainerStarted","Data":"959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297"}
Jul 30 04:37:06.367488 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.367379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k" event={"ID":"33b90a06-0cf1-4193-9eac-760c4742b229","Type":"ContainerStarted","Data":"e804697abd259fd5c777290883926ccde6cf924a7400b5692cf59f5790b01274"}
Jul 30 04:37:06.367488 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.367413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:06.369180 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.369155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc" event={"ID":"badb439f-3412-43a3-9285-25b112eb6f54","Type":"ContainerStarted","Data":"9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552"}
Jul 30 04:37:06.369315 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.369201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc" event={"ID":"badb439f-3412-43a3-9285-25b112eb6f54","Type":"ContainerStarted","Data":"44b04d3c06c9338c8f861debc91638958dbd97b265ea0917a1676c5e56a9e29f"}
Jul 30 04:37:06.369416 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.369376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:06.380870 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.380738    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx" podStartSLOduration=2.380719214 podStartE2EDuration="2.380719214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:37:06.380299693 +0000 UTC m=+3733.334934367" watchObservedRunningTime="2026-07-30 04:37:06.380719214 +0000 UTC m=+3733.335353877"
Jul 30 04:37:06.397839 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.397785    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc" podStartSLOduration=2.048006983 podStartE2EDuration="2.39776939s" totalImagesPullingTime="349.762407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:37:06.396503927 +0000 UTC m=+3733.351138597" watchObservedRunningTime="2026-07-30 04:37:06.39776939 +0000 UTC m=+3733.352404055"
Jul 30 04:37:06.411082 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:06.411025    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k" podStartSLOduration=2.411010689 podStartE2EDuration="2.411010689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:37:06.409810823 +0000 UTC m=+3733.364445491" watchObservedRunningTime="2026-07-30 04:37:06.411010689 +0000 UTC m=+3733.365645359"
Jul 30 04:37:12.385275 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:12.385216    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:37:17.381147 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:17.381116    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:37:17.381610 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:17.381211    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:37:35.022137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:37:35.022102    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx_d70499c2-2753-4155-ad97-504cff67fb8a/payload-processing/0.log"
Jul 30 04:38:29.851806 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:29.851766    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp_ba26bdfb-3f19-4dd1-a037-05518057fc2d/payload-processing/0.log"
Jul 30 04:38:30.078988 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:30.078956    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx_d70499c2-2753-4155-ad97-504cff67fb8a/payload-processing/0.log"
Jul 30 04:38:35.437921 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.437888    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"]
Jul 30 04:38:35.442071 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.442046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:35.443802 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.443786    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-410e1b\""
Jul 30 04:38:35.443876 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.443844    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-410e1b-dockercfg-tpfdf\""
Jul 30 04:38:35.452911 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.452879    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"]
Jul 30 04:38:35.476863 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.476820    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"]
Jul 30 04:38:35.481409 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.481381    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:35.492825 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.492800    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"]
Jul 30 04:38:35.522285 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.522223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052fe3f-cd9e-48db-9b84-c2f083b75406-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2\" (UID: \"6052fe3f-cd9e-48db-9b84-c2f083b75406\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:35.522489 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.522359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21be1676-512e-4630-ac79-486cb0f5aa56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj\" (UID: \"21be1676-512e-4630-ac79-486cb0f5aa56\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:35.522489 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.522398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6tb\" (UniqueName: \"kubernetes.io/projected/6052fe3f-cd9e-48db-9b84-c2f083b75406-kube-api-access-7d6tb\") pod \"payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2\" (UID: \"6052fe3f-cd9e-48db-9b84-c2f083b75406\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:35.522489 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.522464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnxj6\" (UniqueName: \"kubernetes.io/projected/21be1676-512e-4630-ac79-486cb0f5aa56-kube-api-access-tnxj6\") pod \"payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj\" (UID: \"21be1676-512e-4630-ac79-486cb0f5aa56\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:35.623822 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.623778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052fe3f-cd9e-48db-9b84-c2f083b75406-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2\" (UID: \"6052fe3f-cd9e-48db-9b84-c2f083b75406\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:35.624009 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.623935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21be1676-512e-4630-ac79-486cb0f5aa56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj\" (UID: \"21be1676-512e-4630-ac79-486cb0f5aa56\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:35.624009 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.623963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6tb\" (UniqueName: \"kubernetes.io/projected/6052fe3f-cd9e-48db-9b84-c2f083b75406-kube-api-access-7d6tb\") pod \"payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2\" (UID: \"6052fe3f-cd9e-48db-9b84-c2f083b75406\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:35.624009 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.623998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxj6\" (UniqueName: \"kubernetes.io/projected/21be1676-512e-4630-ac79-486cb0f5aa56-kube-api-access-tnxj6\") pod \"payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj\" (UID: \"21be1676-512e-4630-ac79-486cb0f5aa56\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:35.624657 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.624635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052fe3f-cd9e-48db-9b84-c2f083b75406-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2\" (UID: \"6052fe3f-cd9e-48db-9b84-c2f083b75406\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:35.624717 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.624693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21be1676-512e-4630-ac79-486cb0f5aa56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj\" (UID: \"21be1676-512e-4630-ac79-486cb0f5aa56\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:35.631198 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.631174    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxj6\" (UniqueName: \"kubernetes.io/projected/21be1676-512e-4630-ac79-486cb0f5aa56-kube-api-access-tnxj6\") pod \"payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj\" (UID: \"21be1676-512e-4630-ac79-486cb0f5aa56\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:35.631640 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.631619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6tb\" (UniqueName: \"kubernetes.io/projected/6052fe3f-cd9e-48db-9b84-c2f083b75406-kube-api-access-7d6tb\") pod \"payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2\" (UID: \"6052fe3f-cd9e-48db-9b84-c2f083b75406\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:35.753848 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.753757    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:35.793088 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.793045    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:35.922084 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.922047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"]
Jul 30 04:38:35.925202 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:38:35.925153    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6052fe3f_cd9e_48db_9b84_c2f083b75406.slice/crio-ac4f1173da3ec899e685a69d4ac159c6089d67fb557cd7c883bf3f14242a8003 WatchSource:0}: Error finding container ac4f1173da3ec899e685a69d4ac159c6089d67fb557cd7c883bf3f14242a8003: Status 404 returned error can't find the container with id ac4f1173da3ec899e685a69d4ac159c6089d67fb557cd7c883bf3f14242a8003
Jul 30 04:38:35.964503 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:35.964475    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"]
Jul 30 04:38:35.966705 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:38:35.966665    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21be1676_512e_4630_ac79_486cb0f5aa56.slice/crio-ac32436253ec434b607329464f3951f68a38c37aae1c085ec95323b226ac92c1 WatchSource:0}: Error finding container ac32436253ec434b607329464f3951f68a38c37aae1c085ec95323b226ac92c1: Status 404 returned error can't find the container with id ac32436253ec434b607329464f3951f68a38c37aae1c085ec95323b226ac92c1
Jul 30 04:38:36.882566 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:36.882520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj" event={"ID":"21be1676-512e-4630-ac79-486cb0f5aa56","Type":"ContainerStarted","Data":"a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c"}
Jul 30 04:38:36.883080 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:36.882593    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj" event={"ID":"21be1676-512e-4630-ac79-486cb0f5aa56","Type":"ContainerStarted","Data":"ac32436253ec434b607329464f3951f68a38c37aae1c085ec95323b226ac92c1"}
Jul 30 04:38:36.884151 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:36.884121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2" event={"ID":"6052fe3f-cd9e-48db-9b84-c2f083b75406","Type":"ContainerStarted","Data":"5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05"}
Jul 30 04:38:36.884300 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:36.884175    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:36.884300 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:36.884191    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2" event={"ID":"6052fe3f-cd9e-48db-9b84-c2f083b75406","Type":"ContainerStarted","Data":"ac4f1173da3ec899e685a69d4ac159c6089d67fb557cd7c883bf3f14242a8003"}
Jul 30 04:38:36.899685 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:36.899629    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj" podStartSLOduration=1.89961146 podStartE2EDuration="1.89961146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:38:36.8969634 +0000 UTC m=+3823.851598071" watchObservedRunningTime="2026-07-30 04:38:36.89961146 +0000 UTC m=+3823.854246131"
Jul 30 04:38:36.913163 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:36.913108    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2" podStartSLOduration=1.913089756 podStartE2EDuration="1.913089756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:38:36.911056828 +0000 UTC m=+3823.865691497" watchObservedRunningTime="2026-07-30 04:38:36.913089756 +0000 UTC m=+3823.867724422"
Jul 30 04:38:37.890036 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:37.890002    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:38:47.891920 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:47.891888    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:38:48.898019 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:38:48.897980    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:39:15.467147 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.466942    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"]
Jul 30 04:39:15.468672 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.467758    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj" podUID="21be1676-512e-4630-ac79-486cb0f5aa56" containerName="payload-processing" containerID="cri-o://a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c" gracePeriod=30
Jul 30 04:39:15.477567 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.477533    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"]
Jul 30 04:39:15.477851 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.477751    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2" podUID="6052fe3f-cd9e-48db-9b84-c2f083b75406" containerName="payload-pre-processing" containerID="cri-o://5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05" gracePeriod=30
Jul 30 04:39:15.738954 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.738926    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:39:15.772211 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.772182    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:39:15.824665 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.824635    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052fe3f-cd9e-48db-9b84-c2f083b75406-plugins-config-volume\") pod \"6052fe3f-cd9e-48db-9b84-c2f083b75406\" (UID: \"6052fe3f-cd9e-48db-9b84-c2f083b75406\") "
Jul 30 04:39:15.824849 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.824684    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnxj6\" (UniqueName: \"kubernetes.io/projected/21be1676-512e-4630-ac79-486cb0f5aa56-kube-api-access-tnxj6\") pod \"21be1676-512e-4630-ac79-486cb0f5aa56\" (UID: \"21be1676-512e-4630-ac79-486cb0f5aa56\") "
Jul 30 04:39:15.824849 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.824701    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d6tb\" (UniqueName: \"kubernetes.io/projected/6052fe3f-cd9e-48db-9b84-c2f083b75406-kube-api-access-7d6tb\") pod \"6052fe3f-cd9e-48db-9b84-c2f083b75406\" (UID: \"6052fe3f-cd9e-48db-9b84-c2f083b75406\") "
Jul 30 04:39:15.824849 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.824728    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21be1676-512e-4630-ac79-486cb0f5aa56-plugins-config-volume\") pod \"21be1676-512e-4630-ac79-486cb0f5aa56\" (UID: \"21be1676-512e-4630-ac79-486cb0f5aa56\") "
Jul 30 04:39:15.825115 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.825075    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6052fe3f-cd9e-48db-9b84-c2f083b75406-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6052fe3f-cd9e-48db-9b84-c2f083b75406" (UID: "6052fe3f-cd9e-48db-9b84-c2f083b75406"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:39:15.825115 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.825111    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21be1676-512e-4630-ac79-486cb0f5aa56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21be1676-512e-4630-ac79-486cb0f5aa56" (UID: "21be1676-512e-4630-ac79-486cb0f5aa56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:39:15.826755 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.826732    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21be1676-512e-4630-ac79-486cb0f5aa56-kube-api-access-tnxj6" (OuterVolumeSpecName: "kube-api-access-tnxj6") pod "21be1676-512e-4630-ac79-486cb0f5aa56" (UID: "21be1676-512e-4630-ac79-486cb0f5aa56"). InnerVolumeSpecName "kube-api-access-tnxj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:39:15.826853 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.826818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6052fe3f-cd9e-48db-9b84-c2f083b75406-kube-api-access-7d6tb" (OuterVolumeSpecName: "kube-api-access-7d6tb") pod "6052fe3f-cd9e-48db-9b84-c2f083b75406" (UID: "6052fe3f-cd9e-48db-9b84-c2f083b75406"). InnerVolumeSpecName "kube-api-access-7d6tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:39:15.926654 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.926612    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6052fe3f-cd9e-48db-9b84-c2f083b75406-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:15.926654 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.926649    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnxj6\" (UniqueName: \"kubernetes.io/projected/21be1676-512e-4630-ac79-486cb0f5aa56-kube-api-access-tnxj6\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:15.926654 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.926659    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d6tb\" (UniqueName: \"kubernetes.io/projected/6052fe3f-cd9e-48db-9b84-c2f083b75406-kube-api-access-7d6tb\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:15.926917 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:15.926668    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21be1676-512e-4630-ac79-486cb0f5aa56-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:16.111745 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.111652    2560 generic.go:356] "Generic (PLEG): container finished" podID="21be1676-512e-4630-ac79-486cb0f5aa56" containerID="a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c" exitCode=0
Jul 30 04:39:16.111745 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.111694    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"
Jul 30 04:39:16.111990 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.111736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj" event={"ID":"21be1676-512e-4630-ac79-486cb0f5aa56","Type":"ContainerDied","Data":"a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c"}
Jul 30 04:39:16.111990 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.111801    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj" event={"ID":"21be1676-512e-4630-ac79-486cb0f5aa56","Type":"ContainerDied","Data":"ac32436253ec434b607329464f3951f68a38c37aae1c085ec95323b226ac92c1"}
Jul 30 04:39:16.111990 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.111831    2560 scope.go:117] "RemoveContainer" containerID="a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c"
Jul 30 04:39:16.113160 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.113142    2560 generic.go:356] "Generic (PLEG): container finished" podID="6052fe3f-cd9e-48db-9b84-c2f083b75406" containerID="5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05" exitCode=0
Jul 30 04:39:16.113281 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.113190    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"
Jul 30 04:39:16.113281 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.113260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2" event={"ID":"6052fe3f-cd9e-48db-9b84-c2f083b75406","Type":"ContainerDied","Data":"5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05"}
Jul 30 04:39:16.113372 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.113303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2" event={"ID":"6052fe3f-cd9e-48db-9b84-c2f083b75406","Type":"ContainerDied","Data":"ac4f1173da3ec899e685a69d4ac159c6089d67fb557cd7c883bf3f14242a8003"}
Jul 30 04:39:16.127152 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.127124    2560 scope.go:117] "RemoveContainer" containerID="a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c"
Jul 30 04:39:16.127543 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:39:16.127517    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c\": container with ID starting with a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c not found: ID does not exist" containerID="a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c"
Jul 30 04:39:16.127651 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.127556    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c"} err="failed to get container status \"a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c\": rpc error: code = NotFound desc = could not find container \"a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c\": container with ID starting with a6ab42db649bfa9c4fb8fce8b365f3c40470efe510e7c73e6223566313a9890c not found: ID does not exist"
Jul 30 04:39:16.127651 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.127580    2560 scope.go:117] "RemoveContainer" containerID="5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05"
Jul 30 04:39:16.141898 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.141867    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"]
Jul 30 04:39:16.142548 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.142530    2560 scope.go:117] "RemoveContainer" containerID="5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05"
Jul 30 04:39:16.142867 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:39:16.142849    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05\": container with ID starting with 5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05 not found: ID does not exist" containerID="5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05"
Jul 30 04:39:16.142931 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.142879    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05"} err="failed to get container status \"5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05\": rpc error: code = NotFound desc = could not find container \"5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05\": container with ID starting with 5f8b484284b104a7324a45c76a5fb5e837b0b6005e2a7dcce11a59158609ed05 not found: ID does not exist"
Jul 30 04:39:16.144404 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.144384    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-410e1b-c7d58cc9d-9p4gj"]
Jul 30 04:39:16.152857 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.152828    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"]
Jul 30 04:39:16.155717 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:16.155691    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-410e1b-756cc7d96d-6z9v2"]
Jul 30 04:39:17.579374 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:17.579334    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21be1676-512e-4630-ac79-486cb0f5aa56" path="/var/lib/kubelet/pods/21be1676-512e-4630-ac79-486cb0f5aa56/volumes"
Jul 30 04:39:17.579882 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:17.579866    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6052fe3f-cd9e-48db-9b84-c2f083b75406" path="/var/lib/kubelet/pods/6052fe3f-cd9e-48db-9b84-c2f083b75406/volumes"
Jul 30 04:39:25.380216 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.380180    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"]
Jul 30 04:39:25.380830 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.380809    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6052fe3f-cd9e-48db-9b84-c2f083b75406" containerName="payload-pre-processing"
Jul 30 04:39:25.380878 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.380834    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6052fe3f-cd9e-48db-9b84-c2f083b75406" containerName="payload-pre-processing"
Jul 30 04:39:25.380878 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.380850    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21be1676-512e-4630-ac79-486cb0f5aa56" containerName="payload-processing"
Jul 30 04:39:25.380878 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.380859    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="21be1676-512e-4630-ac79-486cb0f5aa56" containerName="payload-processing"
Jul 30 04:39:25.380982 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.380966    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="21be1676-512e-4630-ac79-486cb0f5aa56" containerName="payload-processing"
Jul 30 04:39:25.381016 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.380985    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6052fe3f-cd9e-48db-9b84-c2f083b75406" containerName="payload-pre-processing"
Jul 30 04:39:25.385822 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.385787    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"
Jul 30 04:39:25.388259 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.387943    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:39:25.390989 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.390879    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"]
Jul 30 04:39:25.521546 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.521499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed1bb02-bc89-4081-ba02-f816da34bd25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj\" (UID: \"eed1bb02-bc89-4081-ba02-f816da34bd25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"
Jul 30 04:39:25.622768 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.622731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed1bb02-bc89-4081-ba02-f816da34bd25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj\" (UID: \"eed1bb02-bc89-4081-ba02-f816da34bd25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"
Jul 30 04:39:25.623412 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.623392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed1bb02-bc89-4081-ba02-f816da34bd25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj\" (UID: \"eed1bb02-bc89-4081-ba02-f816da34bd25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"
Jul 30 04:39:25.699834 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.699797    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"
Jul 30 04:39:25.841472 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:25.841443    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"]
Jul 30 04:39:25.843508 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:39:25.843477    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed1bb02_bc89_4081_ba02_f816da34bd25.slice/crio-235c0e21403c103d99e175dc49d8152199c604ee6c915093f62f3f370c94b44f WatchSource:0}: Error finding container 235c0e21403c103d99e175dc49d8152199c604ee6c915093f62f3f370c94b44f: Status 404 returned error can't find the container with id 235c0e21403c103d99e175dc49d8152199c604ee6c915093f62f3f370c94b44f
Jul 30 04:39:26.171880 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:26.171835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj" event={"ID":"eed1bb02-bc89-4081-ba02-f816da34bd25","Type":"ContainerStarted","Data":"235c0e21403c103d99e175dc49d8152199c604ee6c915093f62f3f370c94b44f"}
Jul 30 04:39:27.180301 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:27.180266    2560 generic.go:356] "Generic (PLEG): container finished" podID="eed1bb02-bc89-4081-ba02-f816da34bd25" containerID="0590d67f1e07669fee67c1c5dbcc1d21ece387c811d2b3a82007f477ed017921" exitCode=0
Jul 30 04:39:27.180702 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:27.180398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj" event={"ID":"eed1bb02-bc89-4081-ba02-f816da34bd25","Type":"ContainerDied","Data":"0590d67f1e07669fee67c1c5dbcc1d21ece387c811d2b3a82007f477ed017921"}
Jul 30 04:39:28.337003 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:28.336979    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"
Jul 30 04:39:28.451061 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:28.450960    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed1bb02-bc89-4081-ba02-f816da34bd25-maas-api-service-ca\") pod \"eed1bb02-bc89-4081-ba02-f816da34bd25\" (UID: \"eed1bb02-bc89-4081-ba02-f816da34bd25\") "
Jul 30 04:39:28.451390 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:28.451366    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed1bb02-bc89-4081-ba02-f816da34bd25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eed1bb02-bc89-4081-ba02-f816da34bd25" (UID: "eed1bb02-bc89-4081-ba02-f816da34bd25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:39:28.552033 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:28.551987    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed1bb02-bc89-4081-ba02-f816da34bd25-maas-api-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:29.193811 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:29.193771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj" event={"ID":"eed1bb02-bc89-4081-ba02-f816da34bd25","Type":"ContainerDied","Data":"235c0e21403c103d99e175dc49d8152199c604ee6c915093f62f3f370c94b44f"}
Jul 30 04:39:29.193811 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:29.193809    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"
Jul 30 04:39:29.194034 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:29.193869    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="235c0e21403c103d99e175dc49d8152199c604ee6c915093f62f3f370c94b44f"
Jul 30 04:39:40.864223 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:40.864183    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"]
Jul 30 04:39:40.864730 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:40.864682    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl" podUID="e49bc11f-ddcd-4d12-9900-2456c87de2ac" containerName="maas-api" containerID="cri-o://1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e" gracePeriod=30
Jul 30 04:39:40.929494 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:40.929458    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"]
Jul 30 04:39:40.929844 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:40.929810    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp" podUID="ba26bdfb-3f19-4dd1-a037-05518057fc2d" containerName="payload-processing" containerID="cri-o://31d3f4ced4745c594afd5f834d735258f46614b7ee0d7aa3e49a87a6a30f9f45" gracePeriod=30
Jul 30 04:39:40.942178 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:40.942151    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"]
Jul 30 04:39:40.942477 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:40.942408    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m" podUID="20f2e95e-420a-46e3-b833-3944d81f0214" containerName="payload-pre-processing" containerID="cri-o://427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66" gracePeriod=30
Jul 30 04:39:41.079278 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.078775    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"]
Jul 30 04:39:41.081092 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.081055    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77b2c7-njzgj"]
Jul 30 04:39:41.247020 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.246990    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:39:41.264074 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.264022    2560 generic.go:356] "Generic (PLEG): container finished" podID="ba26bdfb-3f19-4dd1-a037-05518057fc2d" containerID="31d3f4ced4745c594afd5f834d735258f46614b7ee0d7aa3e49a87a6a30f9f45" exitCode=0
Jul 30 04:39:41.264270 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.264085    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp" event={"ID":"ba26bdfb-3f19-4dd1-a037-05518057fc2d","Type":"ContainerDied","Data":"31d3f4ced4745c594afd5f834d735258f46614b7ee0d7aa3e49a87a6a30f9f45"}
Jul 30 04:39:41.264403 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.264318    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:39:41.266439 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.265949    2560 generic.go:356] "Generic (PLEG): container finished" podID="e49bc11f-ddcd-4d12-9900-2456c87de2ac" containerID="1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e" exitCode=0
Jul 30 04:39:41.266439 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.266079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl" event={"ID":"e49bc11f-ddcd-4d12-9900-2456c87de2ac","Type":"ContainerDied","Data":"1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e"}
Jul 30 04:39:41.266439 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.266090    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"
Jul 30 04:39:41.266439 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.266189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl" event={"ID":"e49bc11f-ddcd-4d12-9900-2456c87de2ac","Type":"ContainerDied","Data":"bb9da3361064d61885feecf0652c54320def704ee5e500444724d1b2e18ef4da"}
Jul 30 04:39:41.266439 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.266224    2560 scope.go:117] "RemoveContainer" containerID="1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e"
Jul 30 04:39:41.270051 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.270024    2560 generic.go:356] "Generic (PLEG): container finished" podID="20f2e95e-420a-46e3-b833-3944d81f0214" containerID="427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66" exitCode=0
Jul 30 04:39:41.270165 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.270108    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m" event={"ID":"20f2e95e-420a-46e3-b833-3944d81f0214","Type":"ContainerDied","Data":"427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66"}
Jul 30 04:39:41.270165 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.270148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m" event={"ID":"20f2e95e-420a-46e3-b833-3944d81f0214","Type":"ContainerDied","Data":"6d1e57408f5ce1b58a56c2e1ecad68d9ee7d256a5cc9ca60bcb24f09f978fee9"}
Jul 30 04:39:41.270337 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.270211    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"
Jul 30 04:39:41.279385 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.279353    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvslm\" (UniqueName: \"kubernetes.io/projected/e49bc11f-ddcd-4d12-9900-2456c87de2ac-kube-api-access-gvslm\") pod \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") "
Jul 30 04:39:41.280132 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.279584    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49bc11f-ddcd-4d12-9900-2456c87de2ac-openshift-service-ca\") pod \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") "
Jul 30 04:39:41.280405 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.280375    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49bc11f-ddcd-4d12-9900-2456c87de2ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e49bc11f-ddcd-4d12-9900-2456c87de2ac" (UID: "e49bc11f-ddcd-4d12-9900-2456c87de2ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:39:41.280533 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.280514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls\") pod \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\" (UID: \"e49bc11f-ddcd-4d12-9900-2456c87de2ac\") "
Jul 30 04:39:41.281052 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.281028    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:39:41.281171 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.281041    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49bc11f-ddcd-4d12-9900-2456c87de2ac-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:41.283253 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.283210    2560 scope.go:117] "RemoveContainer" containerID="1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e"
Jul 30 04:39:41.283880 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.283850    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e49bc11f-ddcd-4d12-9900-2456c87de2ac" (UID: "e49bc11f-ddcd-4d12-9900-2456c87de2ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:39:41.285176 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:39:41.285147    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e\": container with ID starting with 1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e not found: ID does not exist" containerID="1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e"
Jul 30 04:39:41.285299 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.285187    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e"} err="failed to get container status \"1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e\": rpc error: code = NotFound desc = could not find container \"1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e\": container with ID starting with 1e1b686bcf8fa6c8ddc897d547364658f87dcc99e7bc728e69e2a4a793c9461e not found: ID does not exist"
Jul 30 04:39:41.285299 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.285213    2560 scope.go:117] "RemoveContainer" containerID="427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66"
Jul 30 04:39:41.285299 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.285284    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49bc11f-ddcd-4d12-9900-2456c87de2ac-kube-api-access-gvslm" (OuterVolumeSpecName: "kube-api-access-gvslm") pod "e49bc11f-ddcd-4d12-9900-2456c87de2ac" (UID: "e49bc11f-ddcd-4d12-9900-2456c87de2ac"). InnerVolumeSpecName "kube-api-access-gvslm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:39:41.294607 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.294587    2560 scope.go:117] "RemoveContainer" containerID="427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66"
Jul 30 04:39:41.294910 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:39:41.294889    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66\": container with ID starting with 427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66 not found: ID does not exist" containerID="427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66"
Jul 30 04:39:41.294971 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.294918    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66"} err="failed to get container status \"427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66\": rpc error: code = NotFound desc = could not find container \"427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66\": container with ID starting with 427040d01e2c8cb30dc1c16e387e68b93e705dbb2a3c4bca0019ac6dbd564b66 not found: ID does not exist"
Jul 30 04:39:41.381970 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.381931    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2bqn\" (UniqueName: \"kubernetes.io/projected/ba26bdfb-3f19-4dd1-a037-05518057fc2d-kube-api-access-l2bqn\") pod \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\" (UID: \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\") "
Jul 30 04:39:41.382186 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.381999    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szk7p\" (UniqueName: \"kubernetes.io/projected/20f2e95e-420a-46e3-b833-3944d81f0214-kube-api-access-szk7p\") pod \"20f2e95e-420a-46e3-b833-3944d81f0214\" (UID: \"20f2e95e-420a-46e3-b833-3944d81f0214\") "
Jul 30 04:39:41.382186 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.382052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f2e95e-420a-46e3-b833-3944d81f0214-plugins-config-volume\") pod \"20f2e95e-420a-46e3-b833-3944d81f0214\" (UID: \"20f2e95e-420a-46e3-b833-3944d81f0214\") "
Jul 30 04:39:41.382367 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.382265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba26bdfb-3f19-4dd1-a037-05518057fc2d-plugins-config-volume\") pod \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\" (UID: \"ba26bdfb-3f19-4dd1-a037-05518057fc2d\") "
Jul 30 04:39:41.382494 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.382471    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f2e95e-420a-46e3-b833-3944d81f0214-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20f2e95e-420a-46e3-b833-3944d81f0214" (UID: "20f2e95e-420a-46e3-b833-3944d81f0214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:39:41.382602 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.382576    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba26bdfb-3f19-4dd1-a037-05518057fc2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba26bdfb-3f19-4dd1-a037-05518057fc2d" (UID: "ba26bdfb-3f19-4dd1-a037-05518057fc2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:39:41.382705 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.382686    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvslm\" (UniqueName: \"kubernetes.io/projected/e49bc11f-ddcd-4d12-9900-2456c87de2ac-kube-api-access-gvslm\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:41.382746 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.382710    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba26bdfb-3f19-4dd1-a037-05518057fc2d-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:41.382746 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.382725    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49bc11f-ddcd-4d12-9900-2456c87de2ac-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:41.382746 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.382738    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f2e95e-420a-46e3-b833-3944d81f0214-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:41.384209 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.384188    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba26bdfb-3f19-4dd1-a037-05518057fc2d-kube-api-access-l2bqn" (OuterVolumeSpecName: "kube-api-access-l2bqn") pod "ba26bdfb-3f19-4dd1-a037-05518057fc2d" (UID: "ba26bdfb-3f19-4dd1-a037-05518057fc2d"). InnerVolumeSpecName "kube-api-access-l2bqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:39:41.384288 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.384253    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f2e95e-420a-46e3-b833-3944d81f0214-kube-api-access-szk7p" (OuterVolumeSpecName: "kube-api-access-szk7p") pod "20f2e95e-420a-46e3-b833-3944d81f0214" (UID: "20f2e95e-420a-46e3-b833-3944d81f0214"). InnerVolumeSpecName "kube-api-access-szk7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:39:41.484285 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.484172    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2bqn\" (UniqueName: \"kubernetes.io/projected/ba26bdfb-3f19-4dd1-a037-05518057fc2d-kube-api-access-l2bqn\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:41.484285 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.484204    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szk7p\" (UniqueName: \"kubernetes.io/projected/20f2e95e-420a-46e3-b833-3944d81f0214-kube-api-access-szk7p\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:39:41.581864 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.581837    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed1bb02-bc89-4081-ba02-f816da34bd25" path="/var/lib/kubelet/pods/eed1bb02-bc89-4081-ba02-f816da34bd25/volumes"
Jul 30 04:39:41.592640 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.592599    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"]
Jul 30 04:39:41.596134 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.596104    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-77b2c7-69d8cb4c5b-vcg5m"]
Jul 30 04:39:41.605726 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.605692    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"]
Jul 30 04:39:41.609974 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:41.609949    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77b2c7-59fdc9f845-txtbl"]
Jul 30 04:39:42.277666 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:42.277576    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"
Jul 30 04:39:42.277666 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:42.277601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp" event={"ID":"ba26bdfb-3f19-4dd1-a037-05518057fc2d","Type":"ContainerDied","Data":"40a8a7feb14f77740dc54bf149900205d81f0a7fa66183a3b9e19fe476fb44be"}
Jul 30 04:39:42.277666 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:42.277664    2560 scope.go:117] "RemoveContainer" containerID="31d3f4ced4745c594afd5f834d735258f46614b7ee0d7aa3e49a87a6a30f9f45"
Jul 30 04:39:42.292394 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:42.292366    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"]
Jul 30 04:39:42.295602 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:42.295566    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-77b2c7-dd7cfd898-rknwp"]
Jul 30 04:39:43.579303 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:43.579268    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20f2e95e-420a-46e3-b833-3944d81f0214" path="/var/lib/kubelet/pods/20f2e95e-420a-46e3-b833-3944d81f0214/volumes"
Jul 30 04:39:43.579700 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:43.579637    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba26bdfb-3f19-4dd1-a037-05518057fc2d" path="/var/lib/kubelet/pods/ba26bdfb-3f19-4dd1-a037-05518057fc2d/volumes"
Jul 30 04:39:43.579984 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:39:43.579971    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e49bc11f-ddcd-4d12-9900-2456c87de2ac" path="/var/lib/kubelet/pods/e49bc11f-ddcd-4d12-9900-2456c87de2ac/volumes"
Jul 30 04:40:35.146904 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.146868    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"]
Jul 30 04:40:35.147539 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.147134    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc" podUID="badb439f-3412-43a3-9285-25b112eb6f54" containerName="maas-api" containerID="cri-o://9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552" gracePeriod=30
Jul 30 04:40:35.199098 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.199062    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"]
Jul 30 04:40:35.199569 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.199532    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx" podUID="d70499c2-2753-4155-ad97-504cff67fb8a" containerName="payload-processing" containerID="cri-o://a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b" gracePeriod=30
Jul 30 04:40:35.206023 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.205958    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"]
Jul 30 04:40:35.206342 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.206193    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k" podUID="33b90a06-0cf1-4193-9eac-760c4742b229" containerName="payload-pre-processing" containerID="cri-o://959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297" gracePeriod=30
Jul 30 04:40:35.465961 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.465933    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:40:35.516542 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.516515    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:40:35.535901 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.535872    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:40:35.581155 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.581112    2560 generic.go:356] "Generic (PLEG): container finished" podID="33b90a06-0cf1-4193-9eac-760c4742b229" containerID="959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297" exitCode=0
Jul 30 04:40:35.581388 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.581204    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"
Jul 30 04:40:35.583670 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.583644    2560 generic.go:356] "Generic (PLEG): container finished" podID="badb439f-3412-43a3-9285-25b112eb6f54" containerID="9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552" exitCode=0
Jul 30 04:40:35.584713 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.584671    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"
Jul 30 04:40:35.587087 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.587064    2560 generic.go:356] "Generic (PLEG): container finished" podID="d70499c2-2753-4155-ad97-504cff67fb8a" containerID="a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b" exitCode=0
Jul 30 04:40:35.587317 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.587175    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"
Jul 30 04:40:35.589885 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.589859    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k" event={"ID":"33b90a06-0cf1-4193-9eac-760c4742b229","Type":"ContainerDied","Data":"959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297"}
Jul 30 04:40:35.589996 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.589913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k" event={"ID":"33b90a06-0cf1-4193-9eac-760c4742b229","Type":"ContainerDied","Data":"e804697abd259fd5c777290883926ccde6cf924a7400b5692cf59f5790b01274"}
Jul 30 04:40:35.589996 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.589935    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc" event={"ID":"badb439f-3412-43a3-9285-25b112eb6f54","Type":"ContainerDied","Data":"9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552"}
Jul 30 04:40:35.589996 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.589954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc" event={"ID":"badb439f-3412-43a3-9285-25b112eb6f54","Type":"ContainerDied","Data":"44b04d3c06c9338c8f861debc91638958dbd97b265ea0917a1676c5e56a9e29f"}
Jul 30 04:40:35.589996 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.589974    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx" event={"ID":"d70499c2-2753-4155-ad97-504cff67fb8a","Type":"ContainerDied","Data":"a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b"}
Jul 30 04:40:35.589996 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.589988    2560 scope.go:117] "RemoveContainer" containerID="959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297"
Jul 30 04:40:35.590312 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.589999    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx" event={"ID":"d70499c2-2753-4155-ad97-504cff67fb8a","Type":"ContainerDied","Data":"7225e72ae35fbd0fe5f5e93d2a88f09f3943c3f91385ef085888010d1b3c1a08"}
Jul 30 04:40:35.599638 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.599619    2560 scope.go:117] "RemoveContainer" containerID="959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297"
Jul 30 04:40:35.599928 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:40:35.599909    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297\": container with ID starting with 959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297 not found: ID does not exist" containerID="959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297"
Jul 30 04:40:35.599982 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.599937    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297"} err="failed to get container status \"959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297\": rpc error: code = NotFound desc = could not find container \"959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297\": container with ID starting with 959b1c1dc510e651af87835f47ff85cd5c6c2c4f83e36afe4e57b50d19898297 not found: ID does not exist"
Jul 30 04:40:35.599982 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.599956    2560 scope.go:117] "RemoveContainer" containerID="9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552"
Jul 30 04:40:35.609950 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.609930    2560 scope.go:117] "RemoveContainer" containerID="9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552"
Jul 30 04:40:35.610215 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:40:35.610196    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552\": container with ID starting with 9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552 not found: ID does not exist" containerID="9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552"
Jul 30 04:40:35.610299 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.610224    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552"} err="failed to get container status \"9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552\": rpc error: code = NotFound desc = could not find container \"9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552\": container with ID starting with 9497d3b49f12f6f3d876ac343e82ea140ca7e28d4eecdc10f4069fb677867552 not found: ID does not exist"
Jul 30 04:40:35.610299 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.610279    2560 scope.go:117] "RemoveContainer" containerID="a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b"
Jul 30 04:40:35.618368 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.618349    2560 scope.go:117] "RemoveContainer" containerID="a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b"
Jul 30 04:40:35.618618 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:40:35.618598    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b\": container with ID starting with a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b not found: ID does not exist" containerID="a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b"
Jul 30 04:40:35.618659 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.618626    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b"} err="failed to get container status \"a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b\": rpc error: code = NotFound desc = could not find container \"a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b\": container with ID starting with a5cbf9282f27b4a2150a6d8236acd6b2c354fd7db129353185d5cbbbc06b616b not found: ID does not exist"
Jul 30 04:40:35.625958 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.625937    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/badb439f-3412-43a3-9285-25b112eb6f54-openshift-service-ca\") pod \"badb439f-3412-43a3-9285-25b112eb6f54\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") "
Jul 30 04:40:35.626041 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.625971    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b90a06-0cf1-4193-9eac-760c4742b229-plugins-config-volume\") pod \"33b90a06-0cf1-4193-9eac-760c4742b229\" (UID: \"33b90a06-0cf1-4193-9eac-760c4742b229\") "
Jul 30 04:40:35.626041 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626032    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgz2p\" (UniqueName: \"kubernetes.io/projected/33b90a06-0cf1-4193-9eac-760c4742b229-kube-api-access-tgz2p\") pod \"33b90a06-0cf1-4193-9eac-760c4742b229\" (UID: \"33b90a06-0cf1-4193-9eac-760c4742b229\") "
Jul 30 04:40:35.626129 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls\") pod \"badb439f-3412-43a3-9285-25b112eb6f54\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") "
Jul 30 04:40:35.626129 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626073    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn99x\" (UniqueName: \"kubernetes.io/projected/badb439f-3412-43a3-9285-25b112eb6f54-kube-api-access-pn99x\") pod \"badb439f-3412-43a3-9285-25b112eb6f54\" (UID: \"badb439f-3412-43a3-9285-25b112eb6f54\") "
Jul 30 04:40:35.626129 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626125    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2grm\" (UniqueName: \"kubernetes.io/projected/d70499c2-2753-4155-ad97-504cff67fb8a-kube-api-access-z2grm\") pod \"d70499c2-2753-4155-ad97-504cff67fb8a\" (UID: \"d70499c2-2753-4155-ad97-504cff67fb8a\") "
Jul 30 04:40:35.626310 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626142    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70499c2-2753-4155-ad97-504cff67fb8a-plugins-config-volume\") pod \"d70499c2-2753-4155-ad97-504cff67fb8a\" (UID: \"d70499c2-2753-4155-ad97-504cff67fb8a\") "
Jul 30 04:40:35.626365 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626313    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badb439f-3412-43a3-9285-25b112eb6f54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "badb439f-3412-43a3-9285-25b112eb6f54" (UID: "badb439f-3412-43a3-9285-25b112eb6f54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:40:35.626365 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626348    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b90a06-0cf1-4193-9eac-760c4742b229-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33b90a06-0cf1-4193-9eac-760c4742b229" (UID: "33b90a06-0cf1-4193-9eac-760c4742b229"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:40:35.626527 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626426    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/badb439f-3412-43a3-9285-25b112eb6f54-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:35.626677 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.626655    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70499c2-2753-4155-ad97-504cff67fb8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d70499c2-2753-4155-ad97-504cff67fb8a" (UID: "d70499c2-2753-4155-ad97-504cff67fb8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:40:35.628204 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.628184    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b90a06-0cf1-4193-9eac-760c4742b229-kube-api-access-tgz2p" (OuterVolumeSpecName: "kube-api-access-tgz2p") pod "33b90a06-0cf1-4193-9eac-760c4742b229" (UID: "33b90a06-0cf1-4193-9eac-760c4742b229"). InnerVolumeSpecName "kube-api-access-tgz2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:40:35.628309 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.628201    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "badb439f-3412-43a3-9285-25b112eb6f54" (UID: "badb439f-3412-43a3-9285-25b112eb6f54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:40:35.628309 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.628256    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70499c2-2753-4155-ad97-504cff67fb8a-kube-api-access-z2grm" (OuterVolumeSpecName: "kube-api-access-z2grm") pod "d70499c2-2753-4155-ad97-504cff67fb8a" (UID: "d70499c2-2753-4155-ad97-504cff67fb8a"). InnerVolumeSpecName "kube-api-access-z2grm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:40:35.628309 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.628265    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badb439f-3412-43a3-9285-25b112eb6f54-kube-api-access-pn99x" (OuterVolumeSpecName: "kube-api-access-pn99x") pod "badb439f-3412-43a3-9285-25b112eb6f54" (UID: "badb439f-3412-43a3-9285-25b112eb6f54"). InnerVolumeSpecName "kube-api-access-pn99x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:40:35.728043 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.727940    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b90a06-0cf1-4193-9eac-760c4742b229-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:35.728043 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.727984    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgz2p\" (UniqueName: \"kubernetes.io/projected/33b90a06-0cf1-4193-9eac-760c4742b229-kube-api-access-tgz2p\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:35.728043 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.728011    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/badb439f-3412-43a3-9285-25b112eb6f54-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:35.728043 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.728026    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn99x\" (UniqueName: \"kubernetes.io/projected/badb439f-3412-43a3-9285-25b112eb6f54-kube-api-access-pn99x\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:35.728043 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.728042    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2grm\" (UniqueName: \"kubernetes.io/projected/d70499c2-2753-4155-ad97-504cff67fb8a-kube-api-access-z2grm\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:35.728477 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.728056    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70499c2-2753-4155-ad97-504cff67fb8a-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:35.903670 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.903641    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"]
Jul 30 04:40:35.909730 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.909707    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1d8c4c-ffc495f88-59g5k"]
Jul 30 04:40:35.918390 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.918365    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"]
Jul 30 04:40:35.922208 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.922182    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d8c4c-8f9b74bf9-94xvc"]
Jul 30 04:40:35.930614 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.930591    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"]
Jul 30 04:40:35.934282 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:35.934259    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1d8c4c-86c6d59df8-s59sx"]
Jul 30 04:40:37.580942 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:37.580910    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b90a06-0cf1-4193-9eac-760c4742b229" path="/var/lib/kubelet/pods/33b90a06-0cf1-4193-9eac-760c4742b229/volumes"
Jul 30 04:40:37.581359 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:37.581345    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="badb439f-3412-43a3-9285-25b112eb6f54" path="/var/lib/kubelet/pods/badb439f-3412-43a3-9285-25b112eb6f54/volumes"
Jul 30 04:40:37.581668 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:37.581657    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d70499c2-2753-4155-ad97-504cff67fb8a" path="/var/lib/kubelet/pods/d70499c2-2753-4155-ad97-504cff67fb8a/volumes"
Jul 30 04:40:52.759792 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:52.759710    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"]
Jul 30 04:40:52.760221 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:52.759965    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd" podUID="21c3bb24-0108-4637-a1f5-40537497c0d0" containerName="maas-api" containerID="cri-o://26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b" gracePeriod=30
Jul 30 04:40:52.776525 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:52.776488    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"]
Jul 30 04:40:52.783020 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:52.782990    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-19ef02-29756430-7smqz"]
Jul 30 04:40:52.808421 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:52.808387    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"]
Jul 30 04:40:52.808852 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:52.808653    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz" podUID="a3d010dc-3398-4dbe-ae0c-921fff0e5f6e" containerName="payload-processing" containerID="cri-o://559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6" gracePeriod=30
Jul 30 04:40:53.098800 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.098772    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:40:53.117174 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.117144    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:40:53.198637 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.198604    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zqkm\" (UniqueName: \"kubernetes.io/projected/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-kube-api-access-6zqkm\") pod \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\" (UID: \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\") "
Jul 30 04:40:53.199103 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.198670    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c3bb24-0108-4637-a1f5-40537497c0d0-maas-api-tls\") pod \"21c3bb24-0108-4637-a1f5-40537497c0d0\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") "
Jul 30 04:40:53.199103 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.198708    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c3bb24-0108-4637-a1f5-40537497c0d0-openshift-service-ca\") pod \"21c3bb24-0108-4637-a1f5-40537497c0d0\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") "
Jul 30 04:40:53.199103 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.198732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6zq8\" (UniqueName: \"kubernetes.io/projected/21c3bb24-0108-4637-a1f5-40537497c0d0-kube-api-access-q6zq8\") pod \"21c3bb24-0108-4637-a1f5-40537497c0d0\" (UID: \"21c3bb24-0108-4637-a1f5-40537497c0d0\") "
Jul 30 04:40:53.199103 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.198777    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-plugins-config-volume\") pod \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\" (UID: \"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e\") "
Jul 30 04:40:53.199408 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.199191    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c3bb24-0108-4637-a1f5-40537497c0d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21c3bb24-0108-4637-a1f5-40537497c0d0" (UID: "21c3bb24-0108-4637-a1f5-40537497c0d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:40:53.199408 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.199285    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3d010dc-3398-4dbe-ae0c-921fff0e5f6e" (UID: "a3d010dc-3398-4dbe-ae0c-921fff0e5f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:40:53.200890 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.200865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c3bb24-0108-4637-a1f5-40537497c0d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21c3bb24-0108-4637-a1f5-40537497c0d0" (UID: "21c3bb24-0108-4637-a1f5-40537497c0d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:40:53.200991 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.200920    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-kube-api-access-6zqkm" (OuterVolumeSpecName: "kube-api-access-6zqkm") pod "a3d010dc-3398-4dbe-ae0c-921fff0e5f6e" (UID: "a3d010dc-3398-4dbe-ae0c-921fff0e5f6e"). InnerVolumeSpecName "kube-api-access-6zqkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:40:53.200991 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.200943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c3bb24-0108-4637-a1f5-40537497c0d0-kube-api-access-q6zq8" (OuterVolumeSpecName: "kube-api-access-q6zq8") pod "21c3bb24-0108-4637-a1f5-40537497c0d0" (UID: "21c3bb24-0108-4637-a1f5-40537497c0d0"). InnerVolumeSpecName "kube-api-access-q6zq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:40:53.300530 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.300446    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqkm\" (UniqueName: \"kubernetes.io/projected/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-kube-api-access-6zqkm\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:53.300530 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.300475    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c3bb24-0108-4637-a1f5-40537497c0d0-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:53.300530 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.300484    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c3bb24-0108-4637-a1f5-40537497c0d0-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:53.300530 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.300493    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6zq8\" (UniqueName: \"kubernetes.io/projected/21c3bb24-0108-4637-a1f5-40537497c0d0-kube-api-access-q6zq8\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:53.300530 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.300505    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:40:53.582316 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.582202    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da710bc-d033-4bc4-96f0-7363e9dca177" path="/var/lib/kubelet/pods/0da710bc-d033-4bc4-96f0-7363e9dca177/volumes"
Jul 30 04:40:53.687538 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.687504    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3d010dc-3398-4dbe-ae0c-921fff0e5f6e" containerID="559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6" exitCode=0
Jul 30 04:40:53.687701 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.687547    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"
Jul 30 04:40:53.687701 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.687592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz" event={"ID":"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e","Type":"ContainerDied","Data":"559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6"}
Jul 30 04:40:53.687701 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.687646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz" event={"ID":"a3d010dc-3398-4dbe-ae0c-921fff0e5f6e","Type":"ContainerDied","Data":"14f42e5bbfe09c1fb392bcfd334510184e4a2aaf462392b630462c5c3fbfad6b"}
Jul 30 04:40:53.687701 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.687677    2560 scope.go:117] "RemoveContainer" containerID="559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6"
Jul 30 04:40:53.689063 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.689043    2560 generic.go:356] "Generic (PLEG): container finished" podID="21c3bb24-0108-4637-a1f5-40537497c0d0" containerID="26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b" exitCode=0
Jul 30 04:40:53.689159 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.689098    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"
Jul 30 04:40:53.689227 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.689101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd" event={"ID":"21c3bb24-0108-4637-a1f5-40537497c0d0","Type":"ContainerDied","Data":"26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b"}
Jul 30 04:40:53.689320 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.689274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd" event={"ID":"21c3bb24-0108-4637-a1f5-40537497c0d0","Type":"ContainerDied","Data":"e77d0d3d1a19d8a0d9c58f0e06d56c43d165b0406815456849a9d500d19c4c8d"}
Jul 30 04:40:53.697172 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.697149    2560 scope.go:117] "RemoveContainer" containerID="559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6"
Jul 30 04:40:53.697534 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:40:53.697510    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6\": container with ID starting with 559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6 not found: ID does not exist" containerID="559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6"
Jul 30 04:40:53.697615 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.697544    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6"} err="failed to get container status \"559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6\": rpc error: code = NotFound desc = could not find container \"559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6\": container with ID starting with 559a9041b8e0c3b52b49085f677a957ba2a0e529c29ab5625e0bc5675b140bc6 not found: ID does not exist"
Jul 30 04:40:53.697615 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.697561    2560 scope.go:117] "RemoveContainer" containerID="26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b"
Jul 30 04:40:53.704994 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.704969    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"]
Jul 30 04:40:53.706388 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.706363    2560 scope.go:117] "RemoveContainer" containerID="26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b"
Jul 30 04:40:53.706718 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:40:53.706680    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b\": container with ID starting with 26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b not found: ID does not exist" containerID="26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b"
Jul 30 04:40:53.706824 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.706716    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b"} err="failed to get container status \"26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b\": rpc error: code = NotFound desc = could not find container \"26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b\": container with ID starting with 26bbccf937779b4a9c24b5073a34a713a66aa4326e38d12d28c91f0991b57a6b not found: ID does not exist"
Jul 30 04:40:53.708879 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.708856    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-19ef02-679d99786-f75qz"]
Jul 30 04:40:53.722003 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.721973    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"]
Jul 30 04:40:53.724028 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:53.724005    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-19ef02-cfdfb854b-grbbd"]
Jul 30 04:40:54.134895 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:54.134860    2560 scope.go:117] "RemoveContainer" containerID="ac0ae7e2b27b1a8f3dfbb5668b9a42fbb856a687a85d4368f4d545c3fc809d05"
Jul 30 04:40:55.580082 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:55.580045    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c3bb24-0108-4637-a1f5-40537497c0d0" path="/var/lib/kubelet/pods/21c3bb24-0108-4637-a1f5-40537497c0d0/volumes"
Jul 30 04:40:55.580477 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:55.580466    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d010dc-3398-4dbe-ae0c-921fff0e5f6e" path="/var/lib/kubelet/pods/a3d010dc-3398-4dbe-ae0c-921fff0e5f6e/volumes"
Jul 30 04:40:59.775379 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775342    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"]
Jul 30 04:40:59.775763 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775750    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c3bb24-0108-4637-a1f5-40537497c0d0" containerName="maas-api"
Jul 30 04:40:59.775763 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775764    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c3bb24-0108-4637-a1f5-40537497c0d0" containerName="maas-api"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775780    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f2e95e-420a-46e3-b833-3944d81f0214" containerName="payload-pre-processing"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775786    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f2e95e-420a-46e3-b833-3944d81f0214" containerName="payload-pre-processing"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775794    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba26bdfb-3f19-4dd1-a037-05518057fc2d" containerName="payload-processing"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775800    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba26bdfb-3f19-4dd1-a037-05518057fc2d" containerName="payload-processing"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775807    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70499c2-2753-4155-ad97-504cff67fb8a" containerName="payload-processing"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775812    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70499c2-2753-4155-ad97-504cff67fb8a" containerName="payload-processing"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775818    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="badb439f-3412-43a3-9285-25b112eb6f54" containerName="maas-api"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775823    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="badb439f-3412-43a3-9285-25b112eb6f54" containerName="maas-api"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775834    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d010dc-3398-4dbe-ae0c-921fff0e5f6e" containerName="payload-processing"
Jul 30 04:40:59.775838 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775838    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d010dc-3398-4dbe-ae0c-921fff0e5f6e" containerName="payload-processing"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775844    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b90a06-0cf1-4193-9eac-760c4742b229" containerName="payload-pre-processing"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775851    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b90a06-0cf1-4193-9eac-760c4742b229" containerName="payload-pre-processing"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775857    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed1bb02-bc89-4081-ba02-f816da34bd25" containerName="revoke-keys"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775863    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed1bb02-bc89-4081-ba02-f816da34bd25" containerName="revoke-keys"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775873    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49bc11f-ddcd-4d12-9900-2456c87de2ac" containerName="maas-api"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775879    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49bc11f-ddcd-4d12-9900-2456c87de2ac" containerName="maas-api"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775939    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed1bb02-bc89-4081-ba02-f816da34bd25" containerName="revoke-keys"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775948    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d010dc-3398-4dbe-ae0c-921fff0e5f6e" containerName="payload-processing"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775955    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d70499c2-2753-4155-ad97-504cff67fb8a" containerName="payload-processing"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775961    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f2e95e-420a-46e3-b833-3944d81f0214" containerName="payload-pre-processing"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775966    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49bc11f-ddcd-4d12-9900-2456c87de2ac" containerName="maas-api"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775972    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c3bb24-0108-4637-a1f5-40537497c0d0" containerName="maas-api"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775977    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="badb439f-3412-43a3-9285-25b112eb6f54" containerName="maas-api"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775983    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b90a06-0cf1-4193-9eac-760c4742b229" containerName="payload-pre-processing"
Jul 30 04:40:59.776137 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.775989    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba26bdfb-3f19-4dd1-a037-05518057fc2d" containerName="payload-processing"
Jul 30 04:40:59.780743 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.780716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"
Jul 30 04:40:59.783369 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.782889    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5cg4s\""
Jul 30 04:40:59.785456 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.785432    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"]
Jul 30 04:40:59.864706 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.864669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/565965f3-6466-4e8d-a0d3-d9f61ce437c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv\" (UID: \"565965f3-6466-4e8d-a0d3-d9f61ce437c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"
Jul 30 04:40:59.966019 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.965985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/565965f3-6466-4e8d-a0d3-d9f61ce437c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv\" (UID: \"565965f3-6466-4e8d-a0d3-d9f61ce437c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"
Jul 30 04:40:59.966707 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:40:59.966686    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/565965f3-6466-4e8d-a0d3-d9f61ce437c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv\" (UID: \"565965f3-6466-4e8d-a0d3-d9f61ce437c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"
Jul 30 04:41:00.094950 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:00.094859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"
Jul 30 04:41:00.236029 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:00.235994    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"]
Jul 30 04:41:00.239338 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:41:00.239305    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565965f3_6466_4e8d_a0d3_d9f61ce437c4.slice/crio-9205f89369581a8080fb0b1ea816742ca163e33af5e49c6862b9e00ba45bd47b WatchSource:0}: Error finding container 9205f89369581a8080fb0b1ea816742ca163e33af5e49c6862b9e00ba45bd47b: Status 404 returned error can't find the container with id 9205f89369581a8080fb0b1ea816742ca163e33af5e49c6862b9e00ba45bd47b
Jul 30 04:41:00.727062 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:00.727019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv" event={"ID":"565965f3-6466-4e8d-a0d3-d9f61ce437c4","Type":"ContainerStarted","Data":"932f71a17adf2f227c1cb8cc549eff3b4b49ab82a04001ab006632d38e83e64c"}
Jul 30 04:41:00.727208 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:00.727087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv" event={"ID":"565965f3-6466-4e8d-a0d3-d9f61ce437c4","Type":"ContainerStarted","Data":"9205f89369581a8080fb0b1ea816742ca163e33af5e49c6862b9e00ba45bd47b"}
Jul 30 04:41:00.740570 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:00.740468    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv" podStartSLOduration=1.7404485539999999 podStartE2EDuration="1.740448554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:41:00.739747828 +0000 UTC m=+3967.694382501" watchObservedRunningTime="2026-07-30 04:41:00.740448554 +0000 UTC m=+3967.695083224"
Jul 30 04:41:01.733975 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:01.733938    2560 generic.go:356] "Generic (PLEG): container finished" podID="565965f3-6466-4e8d-a0d3-d9f61ce437c4" containerID="932f71a17adf2f227c1cb8cc549eff3b4b49ab82a04001ab006632d38e83e64c" exitCode=0
Jul 30 04:41:01.734381 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:01.734007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv" event={"ID":"565965f3-6466-4e8d-a0d3-d9f61ce437c4","Type":"ContainerDied","Data":"932f71a17adf2f227c1cb8cc549eff3b4b49ab82a04001ab006632d38e83e64c"}
Jul 30 04:41:02.882206 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:02.882175    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"
Jul 30 04:41:02.996286 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:02.996168    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/565965f3-6466-4e8d-a0d3-d9f61ce437c4-maas-api-service-ca\") pod \"565965f3-6466-4e8d-a0d3-d9f61ce437c4\" (UID: \"565965f3-6466-4e8d-a0d3-d9f61ce437c4\") "
Jul 30 04:41:02.996565 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:02.996543    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565965f3-6466-4e8d-a0d3-d9f61ce437c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "565965f3-6466-4e8d-a0d3-d9f61ce437c4" (UID: "565965f3-6466-4e8d-a0d3-d9f61ce437c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:41:03.097296 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:03.097264    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/565965f3-6466-4e8d-a0d3-d9f61ce437c4-maas-api-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:41:03.746923 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:03.746880    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv" event={"ID":"565965f3-6466-4e8d-a0d3-d9f61ce437c4","Type":"ContainerDied","Data":"9205f89369581a8080fb0b1ea816742ca163e33af5e49c6862b9e00ba45bd47b"}
Jul 30 04:41:03.746923 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:03.746917    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"
Jul 30 04:41:03.747135 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:03.746945    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9205f89369581a8080fb0b1ea816742ca163e33af5e49c6862b9e00ba45bd47b"
Jul 30 04:41:10.208862 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.208829    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"]
Jul 30 04:41:10.209519 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.209089    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j" podUID="0b1b0b09-a629-4aff-8048-07ff201a3575" containerName="maas-api" containerID="cri-o://71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd" gracePeriod=30
Jul 30 04:41:10.238834 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.238805    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"]
Jul 30 04:41:10.242584 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.242556    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-fcf3e6-29756430-fwh2p"]
Jul 30 04:41:10.277410 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.277377    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"]
Jul 30 04:41:10.277681 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.277637    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485" podUID="81c08e06-0896-4db5-a9e7-ea9644e47060" containerName="payload-pre-processing" containerID="cri-o://47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931" gracePeriod=30
Jul 30 04:41:10.411247 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.411192    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"]
Jul 30 04:41:10.414695 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.414667    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcf3e6-r59pv"]
Jul 30 04:41:10.532865 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.532835    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:41:10.568036 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.568008    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:41:10.671376 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.671342    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1b0b09-a629-4aff-8048-07ff201a3575-openshift-service-ca\") pod \"0b1b0b09-a629-4aff-8048-07ff201a3575\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") "
Jul 30 04:41:10.671376 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.671379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c08e06-0896-4db5-a9e7-ea9644e47060-plugins-config-volume\") pod \"81c08e06-0896-4db5-a9e7-ea9644e47060\" (UID: \"81c08e06-0896-4db5-a9e7-ea9644e47060\") "
Jul 30 04:41:10.671640 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.671463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls\") pod \"0b1b0b09-a629-4aff-8048-07ff201a3575\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") "
Jul 30 04:41:10.671640 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.671505    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml6md\" (UniqueName: \"kubernetes.io/projected/0b1b0b09-a629-4aff-8048-07ff201a3575-kube-api-access-ml6md\") pod \"0b1b0b09-a629-4aff-8048-07ff201a3575\" (UID: \"0b1b0b09-a629-4aff-8048-07ff201a3575\") "
Jul 30 04:41:10.671640 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.671553    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvx65\" (UniqueName: \"kubernetes.io/projected/81c08e06-0896-4db5-a9e7-ea9644e47060-kube-api-access-vvx65\") pod \"81c08e06-0896-4db5-a9e7-ea9644e47060\" (UID: \"81c08e06-0896-4db5-a9e7-ea9644e47060\") "
Jul 30 04:41:10.671845 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.671816    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1b0b09-a629-4aff-8048-07ff201a3575-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b1b0b09-a629-4aff-8048-07ff201a3575" (UID: "0b1b0b09-a629-4aff-8048-07ff201a3575"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:41:10.672016 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.671863    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c08e06-0896-4db5-a9e7-ea9644e47060-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81c08e06-0896-4db5-a9e7-ea9644e47060" (UID: "81c08e06-0896-4db5-a9e7-ea9644e47060"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:41:10.673722 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.673689    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b1b0b09-a629-4aff-8048-07ff201a3575" (UID: "0b1b0b09-a629-4aff-8048-07ff201a3575"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:41:10.673814 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.673780    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1b0b09-a629-4aff-8048-07ff201a3575-kube-api-access-ml6md" (OuterVolumeSpecName: "kube-api-access-ml6md") pod "0b1b0b09-a629-4aff-8048-07ff201a3575" (UID: "0b1b0b09-a629-4aff-8048-07ff201a3575"). InnerVolumeSpecName "kube-api-access-ml6md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:41:10.673948 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.673928    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c08e06-0896-4db5-a9e7-ea9644e47060-kube-api-access-vvx65" (OuterVolumeSpecName: "kube-api-access-vvx65") pod "81c08e06-0896-4db5-a9e7-ea9644e47060" (UID: "81c08e06-0896-4db5-a9e7-ea9644e47060"). InnerVolumeSpecName "kube-api-access-vvx65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:41:10.772451 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.772368    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1b0b09-a629-4aff-8048-07ff201a3575-maas-api-tls\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:41:10.772451 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.772398    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml6md\" (UniqueName: \"kubernetes.io/projected/0b1b0b09-a629-4aff-8048-07ff201a3575-kube-api-access-ml6md\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:41:10.772451 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.772408    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvx65\" (UniqueName: \"kubernetes.io/projected/81c08e06-0896-4db5-a9e7-ea9644e47060-kube-api-access-vvx65\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:41:10.772451 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.772417    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1b0b09-a629-4aff-8048-07ff201a3575-openshift-service-ca\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:41:10.772451 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.772426    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c08e06-0896-4db5-a9e7-ea9644e47060-plugins-config-volume\") on node \"ip-10-0-137-125.ec2.internal\" DevicePath \"\""
Jul 30 04:41:10.783881 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.783850    2560 generic.go:356] "Generic (PLEG): container finished" podID="81c08e06-0896-4db5-a9e7-ea9644e47060" containerID="47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931" exitCode=0
Jul 30 04:41:10.784028 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.783910    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"
Jul 30 04:41:10.784028 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.783938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485" event={"ID":"81c08e06-0896-4db5-a9e7-ea9644e47060","Type":"ContainerDied","Data":"47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931"}
Jul 30 04:41:10.784028 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.783995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485" event={"ID":"81c08e06-0896-4db5-a9e7-ea9644e47060","Type":"ContainerDied","Data":"1d54fde44feadf8529a7903f872fda403710498b033edea851b2c535c96ce554"}
Jul 30 04:41:10.784028 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.784019    2560 scope.go:117] "RemoveContainer" containerID="47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931"
Jul 30 04:41:10.785489 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.785468    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b1b0b09-a629-4aff-8048-07ff201a3575" containerID="71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd" exitCode=0
Jul 30 04:41:10.785604 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.785530    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j" event={"ID":"0b1b0b09-a629-4aff-8048-07ff201a3575","Type":"ContainerDied","Data":"71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd"}
Jul 30 04:41:10.785604 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.785546    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"
Jul 30 04:41:10.785604 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.785591    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j" event={"ID":"0b1b0b09-a629-4aff-8048-07ff201a3575","Type":"ContainerDied","Data":"09ad2fdda0b049ca9edcf1cbf53ecd5d78b61fa6fbd545a9b45d6710c7788f3f"}
Jul 30 04:41:10.794543 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.794521    2560 scope.go:117] "RemoveContainer" containerID="47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931"
Jul 30 04:41:10.794845 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:41:10.794823    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931\": container with ID starting with 47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931 not found: ID does not exist" containerID="47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931"
Jul 30 04:41:10.794910 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.794854    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931"} err="failed to get container status \"47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931\": rpc error: code = NotFound desc = could not find container \"47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931\": container with ID starting with 47e615d5f44e81af0c65252ba1fe01c3342a093b5ee0dc2d363a02814667f931 not found: ID does not exist"
Jul 30 04:41:10.794910 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.794871    2560 scope.go:117] "RemoveContainer" containerID="71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd"
Jul 30 04:41:10.803760 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.803741    2560 scope.go:117] "RemoveContainer" containerID="71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd"
Jul 30 04:41:10.804009 ip-10-0-137-125 kubenswrapper[2560]: E0730 04:41:10.803992    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd\": container with ID starting with 71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd not found: ID does not exist" containerID="71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd"
Jul 30 04:41:10.804063 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.804016    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd"} err="failed to get container status \"71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd\": rpc error: code = NotFound desc = could not find container \"71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd\": container with ID starting with 71a6c1decd1df0d9e248934c027963905c956f82bcca9e07ab289bc57fb1bedd not found: ID does not exist"
Jul 30 04:41:10.809040 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.809017    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"]
Jul 30 04:41:10.813155 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.813131    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fcf3e6-667f48b876-jq485"]
Jul 30 04:41:10.823086 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.823061    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"]
Jul 30 04:41:10.826937 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:10.826912    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fcf3e6-77dcc6cb9c-2lz8j"]
Jul 30 04:41:11.581472 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:11.581389    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1b0b09-a629-4aff-8048-07ff201a3575" path="/var/lib/kubelet/pods/0b1b0b09-a629-4aff-8048-07ff201a3575/volumes"
Jul 30 04:41:11.581958 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:11.581764    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="120b7df7-e763-48a3-b3a7-339c06379452" path="/var/lib/kubelet/pods/120b7df7-e763-48a3-b3a7-339c06379452/volumes"
Jul 30 04:41:11.582126 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:11.582108    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="565965f3-6466-4e8d-a0d3-d9f61ce437c4" path="/var/lib/kubelet/pods/565965f3-6466-4e8d-a0d3-d9f61ce437c4/volumes"
Jul 30 04:41:11.582497 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:11.582479    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c08e06-0896-4db5-a9e7-ea9644e47060" path="/var/lib/kubelet/pods/81c08e06-0896-4db5-a9e7-ea9644e47060/volumes"
Jul 30 04:41:22.608777 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:22.608744    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qwpsf_5f36ccbe-8559-491d-840d-6ad6061a55dc/manager/0.log"
Jul 30 04:41:22.722643 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:22.722610    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9bc959cf-b4khk_1303e528-ea13-4f48-9a96-7c52aa717057/manager/0.log"
Jul 30 04:41:23.271775 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:23.271739    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5946c6c8fb-xnvfc_d585fc93-cf2b-4348-86b0-1b3ff8907f78/copy-manifests/0.log"
Jul 30 04:41:23.290217 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:23.290185    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5946c6c8fb-xnvfc_d585fc93-cf2b-4348-86b0-1b3ff8907f78/manager/0.log"
Jul 30 04:41:23.617803 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:23.617700    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-95d5b8d55-vknqp_20a7f08d-1113-4c31-837f-0db8f106a7dd/maas-api/0.log"
Jul 30 04:41:23.721192 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:23.721156    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756400-5btw9_be2e0ebe-2cfd-4657-9f2c-8ef6e2f996a6/cleanup/0.log"
Jul 30 04:41:23.827995 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:23.827967    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756415-9hh6s_7bcf3868-f8fd-4d7a-9e18-ed85fb429a91/cleanup/0.log"
Jul 30 04:41:23.932322 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:23.932292    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756430-zk6b2_678b6af3-0869-43bf-8b47-5a9784e8c97c/cleanup/0.log"
Jul 30 04:41:25.810076 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:25.810039    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rxm9m_877738e4-a61e-40ee-bef9-eb64606048c5/manager/0.log"
Jul 30 04:41:26.036359 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:26.036323    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-957ss_6f3a62dd-136a-4929-8670-d4b7a90a1a43/manager/0.log"
Jul 30 04:41:26.904457 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:26.904425    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-vpknv_6678804d-3999-4dca-a570-dce6d84c1208/payload-pre-processing/0.log"
Jul 30 04:41:27.550967 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:27.550918    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-847mw_fd5bbe32-5507-4d1d-a2c8-b1298ed32085/main/0.log"
Jul 30 04:41:27.753302 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:27.753261    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb2wc9_0f297d50-dd9d-4da9-9e79-86f45e15f023/main/0.log"
Jul 30 04:41:27.854504 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:27.854410    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-c9vj2_5e93e0ca-19c0-4d3a-b385-18b3b305bc4a/main/0.log"
Jul 30 04:41:35.891688 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:35.891656    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xfcs5_239554d7-aed4-4f52-8b1e-fdd8b3758c97/global-pull-secret-syncer/0.log"
Jul 30 04:41:35.989289 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:35.989257    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r5szt_1236a4cf-a822-4222-ab7b-ecec2387f240/konnectivity-agent/0.log"
Jul 30 04:41:36.082405 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:36.082367    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-125.ec2.internal_8d00ccf9cbba644f9ba54a54e77d171e/haproxy/0.log"
Jul 30 04:41:41.157307 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:41.157274    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rxm9m_877738e4-a61e-40ee-bef9-eb64606048c5/manager/0.log"
Jul 30 04:41:41.305683 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:41.305599    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-957ss_6f3a62dd-136a-4929-8670-d4b7a90a1a43/manager/0.log"
Jul 30 04:41:43.198252 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.198199    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-llplc_fbc36ada-4a95-4b5a-a34f-ca97991de624/kube-state-metrics/0.log"
Jul 30 04:41:43.216184 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.216150    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-llplc_fbc36ada-4a95-4b5a-a34f-ca97991de624/kube-rbac-proxy-main/0.log"
Jul 30 04:41:43.235556 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.235523    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-llplc_fbc36ada-4a95-4b5a-a34f-ca97991de624/kube-rbac-proxy-self/0.log"
Jul 30 04:41:43.259112 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.259084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c57d6c8-bbgsg_45cc857f-c973-46f1-8cb3-7ae3efe45767/metrics-server/0.log"
Jul 30 04:41:43.281782 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.281756    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-kdm4d_f99ec3fe-a2c1-441a-8361-3f8f7fdcb886/monitoring-plugin/0.log"
Jul 30 04:41:43.526158 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.526062    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s42w5_991402de-3dde-4b38-8878-d54c9eae6f9f/node-exporter/0.log"
Jul 30 04:41:43.544005 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.543982    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s42w5_991402de-3dde-4b38-8878-d54c9eae6f9f/kube-rbac-proxy/0.log"
Jul 30 04:41:43.569955 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.569917    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s42w5_991402de-3dde-4b38-8878-d54c9eae6f9f/init-textfile/0.log"
Jul 30 04:41:43.594853 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.594814    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-sv77j_5be24e71-72df-46b3-8aff-844f4355f31a/kube-rbac-proxy-main/0.log"
Jul 30 04:41:43.615124 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.615078    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-sv77j_5be24e71-72df-46b3-8aff-844f4355f31a/kube-rbac-proxy-self/0.log"
Jul 30 04:41:43.636086 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.636047    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-sv77j_5be24e71-72df-46b3-8aff-844f4355f31a/openshift-state-metrics/0.log"
Jul 30 04:41:43.679075 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.679051    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1683baa-26b5-4cde-a961-0404511036e1/prometheus/0.log"
Jul 30 04:41:43.701588 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.701561    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1683baa-26b5-4cde-a961-0404511036e1/config-reloader/0.log"
Jul 30 04:41:43.721166 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.721136    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1683baa-26b5-4cde-a961-0404511036e1/thanos-sidecar/0.log"
Jul 30 04:41:43.741797 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.741766    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1683baa-26b5-4cde-a961-0404511036e1/kube-rbac-proxy-web/0.log"
Jul 30 04:41:43.759614 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.759585    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1683baa-26b5-4cde-a961-0404511036e1/kube-rbac-proxy/0.log"
Jul 30 04:41:43.780374 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.780309    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1683baa-26b5-4cde-a961-0404511036e1/kube-rbac-proxy-thanos/0.log"
Jul 30 04:41:43.801245 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.801207    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1683baa-26b5-4cde-a961-0404511036e1/init-config-reloader/0.log"
Jul 30 04:41:43.877505 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.877472    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-rpzcc_9236e3ee-cc6a-4a67-ad5d-0fa156d706b6/prometheus-operator-admission-webhook/0.log"
Jul 30 04:41:43.903197 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.903148    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5557d7d88f-dljgh_76f62d5f-1f8f-43b8-95d0-950a657878ef/telemeter-client/0.log"
Jul 30 04:41:43.922521 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.922499    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5557d7d88f-dljgh_76f62d5f-1f8f-43b8-95d0-950a657878ef/reload/0.log"
Jul 30 04:41:43.940760 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:43.940738    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5557d7d88f-dljgh_76f62d5f-1f8f-43b8-95d0-950a657878ef/kube-rbac-proxy/0.log"
Jul 30 04:41:46.015436 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015402    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"]
Jul 30 04:41:46.015840 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015815    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1b0b09-a629-4aff-8048-07ff201a3575" containerName="maas-api"
Jul 30 04:41:46.015840 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015828    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1b0b09-a629-4aff-8048-07ff201a3575" containerName="maas-api"
Jul 30 04:41:46.015840 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015836    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565965f3-6466-4e8d-a0d3-d9f61ce437c4" containerName="revoke-keys"
Jul 30 04:41:46.015965 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015844    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="565965f3-6466-4e8d-a0d3-d9f61ce437c4" containerName="revoke-keys"
Jul 30 04:41:46.015965 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015866    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c08e06-0896-4db5-a9e7-ea9644e47060" containerName="payload-pre-processing"
Jul 30 04:41:46.015965 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015875    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c08e06-0896-4db5-a9e7-ea9644e47060" containerName="payload-pre-processing"
Jul 30 04:41:46.015965 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015961    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1b0b09-a629-4aff-8048-07ff201a3575" containerName="maas-api"
Jul 30 04:41:46.016105 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015975    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="565965f3-6466-4e8d-a0d3-d9f61ce437c4" containerName="revoke-keys"
Jul 30 04:41:46.016105 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.015985    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c08e06-0896-4db5-a9e7-ea9644e47060" containerName="payload-pre-processing"
Jul 30 04:41:46.019295 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.019277    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.021650 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.021627    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ckfp2\"/\"kube-root-ca.crt\""
Jul 30 04:41:46.021929 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.021913    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ckfp2\"/\"openshift-service-ca.crt\""
Jul 30 04:41:46.022017 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.021926    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ckfp2\"/\"default-dockercfg-pr8bt\""
Jul 30 04:41:46.027884 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.027862    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"]
Jul 30 04:41:46.117904 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.117865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-podres\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.118089 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.117922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vbv\" (UniqueName: \"kubernetes.io/projected/38f1cf79-a45b-4057-876f-6c8deeedf35e-kube-api-access-89vbv\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.118089 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.118003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-proc\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.118171 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.118115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-sys\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.118171 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.118162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-lib-modules\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.218903 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.218859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vbv\" (UniqueName: \"kubernetes.io/projected/38f1cf79-a45b-4057-876f-6c8deeedf35e-kube-api-access-89vbv\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.219109 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.218920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-proc\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.219109 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.218991    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-sys\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.219109 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.219029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-lib-modules\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.219109 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.219062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-podres\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.219109 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.219065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-proc\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.219384 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.219117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-sys\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.219384 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.219211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-podres\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.219384 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.219210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38f1cf79-a45b-4057-876f-6c8deeedf35e-lib-modules\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.226036 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.226019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vbv\" (UniqueName: \"kubernetes.io/projected/38f1cf79-a45b-4057-876f-6c8deeedf35e-kube-api-access-89vbv\") pod \"perf-node-gather-daemonset-2nzst\" (UID: \"38f1cf79-a45b-4057-876f-6c8deeedf35e\") " pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.330847 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.330747    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:46.467426 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.467400    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"]
Jul 30 04:41:46.469905 ip-10-0-137-125 kubenswrapper[2560]: W0730 04:41:46.469870    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38f1cf79_a45b_4057_876f_6c8deeedf35e.slice/crio-9973e47a9a7e952234710946ca19456c42a7721de23cfc417bc29c9a0eab2c91 WatchSource:0}: Error finding container 9973e47a9a7e952234710946ca19456c42a7721de23cfc417bc29c9a0eab2c91: Status 404 returned error can't find the container with id 9973e47a9a7e952234710946ca19456c42a7721de23cfc417bc29c9a0eab2c91
Jul 30 04:41:46.982121 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.982081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst" event={"ID":"38f1cf79-a45b-4057-876f-6c8deeedf35e","Type":"ContainerStarted","Data":"38fe3890dafc83dff3f10365bc801fc0369b80a1ec1d96b1f90c149c29ba487e"}
Jul 30 04:41:46.982294 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.982138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst" event={"ID":"38f1cf79-a45b-4057-876f-6c8deeedf35e","Type":"ContainerStarted","Data":"9973e47a9a7e952234710946ca19456c42a7721de23cfc417bc29c9a0eab2c91"}
Jul 30 04:41:46.982294 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:46.982199    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:47.020035 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:47.019975    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst" podStartSLOduration=2.019955953 podStartE2EDuration="2.019955953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 04:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:41:47.019297332 +0000 UTC m=+4013.973932003" watchObservedRunningTime="2026-07-30 04:41:47.019955953 +0000 UTC m=+4013.974590624"
Jul 30 04:41:47.082931 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:47.082899    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-vx4rn_a1e976d1-c889-442b-834d-fd69e9bca652/volume-data-source-validator/0.log"
Jul 30 04:41:48.082155 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:48.082118    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7t5ct_d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa/dns/0.log"
Jul 30 04:41:48.099490 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:48.099468    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7t5ct_d1c014a2-8b6d-4325-b3f4-f3d0c6534cfa/kube-rbac-proxy/0.log"
Jul 30 04:41:48.237363 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:48.237335    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fltkz_48c8e946-a078-4b39-912c-fde7009abb28/dns-node-resolver/0.log"
Jul 30 04:41:48.798792 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:48.798757    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5879b768bb-n7g4g_d4565b6b-c6ff-474b-8208-b83400bce0c4/registry/0.log"
Jul 30 04:41:48.845865 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:48.845827    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p2q9q_519e4a13-90e3-4d40-a097-cecb5171c54c/node-ca/0.log"
Jul 30 04:41:50.180832 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:50.180797    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-vpknv_6678804d-3999-4dca-a570-dce6d84c1208/payload-pre-processing/0.log"
Jul 30 04:41:51.060657 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:51.060624    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ctdpt_58a0e78a-b220-4eb2-8435-7298c7e001f4/serve-healthcheck-canary/0.log"
Jul 30 04:41:51.721607 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:51.721571    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xdwvl_2a796f82-3873-4cc3-92ee-b6bc2b80240a/kube-rbac-proxy/0.log"
Jul 30 04:41:51.742445 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:51.742412    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xdwvl_2a796f82-3873-4cc3-92ee-b6bc2b80240a/exporter/0.log"
Jul 30 04:41:51.767997 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:51.767968    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xdwvl_2a796f82-3873-4cc3-92ee-b6bc2b80240a/extractor/0.log"
Jul 30 04:41:52.997812 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:52.997783    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ckfp2/perf-node-gather-daemonset-2nzst"
Jul 30 04:41:54.175561 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:54.175530    2560 scope.go:117] "RemoveContainer" containerID="64477f2893da7761e25c20347c5ff0cf33c58b601b41670ae2d45cd6a476d728"
Jul 30 04:41:54.216088 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:54.216063    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qwpsf_5f36ccbe-8559-491d-840d-6ad6061a55dc/manager/0.log"
Jul 30 04:41:54.469102 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:54.469010    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9bc959cf-b4khk_1303e528-ea13-4f48-9a96-7c52aa717057/manager/0.log"
Jul 30 04:41:54.849407 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:54.849378    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5946c6c8fb-xnvfc_d585fc93-cf2b-4348-86b0-1b3ff8907f78/manager/0.log"
Jul 30 04:41:54.869793 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:54.869764    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5946c6c8fb-xnvfc_d585fc93-cf2b-4348-86b0-1b3ff8907f78/copy-manifests/0.log"
Jul 30 04:41:56.298091 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:56.298007    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bb6c44f89-hlv5b_32d44a91-d63a-4e0c-837b-afff21107c7f/manager/0.log"
Jul 30 04:41:56.323815 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:56.323765    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ttp2s_9ca16d7c-efb9-4f48-b81c-2e5256771c7e/openshift-lws-operator/0.log"
Jul 30 04:41:56.564962 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:56.564879    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-vpknv_6678804d-3999-4dca-a570-dce6d84c1208/payload-pre-processing/0.log"
Jul 30 04:41:58.297361 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:41:58.297328    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-vpknv_6678804d-3999-4dca-a570-dce6d84c1208/payload-pre-processing/0.log"
Jul 30 04:42:02.214816 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.214783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-22d4x_f8708fd8-0e95-41fb-914c-54b8f9828bd4/kube-multus/0.log"
Jul 30 04:42:02.516347 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.516315    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mlpxh_30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd/kube-multus-additional-cni-plugins/0.log"
Jul 30 04:42:02.537992 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.537960    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mlpxh_30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd/egress-router-binary-copy/0.log"
Jul 30 04:42:02.560122 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.560096    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mlpxh_30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd/cni-plugins/0.log"
Jul 30 04:42:02.577303 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.577264    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mlpxh_30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd/bond-cni-plugin/0.log"
Jul 30 04:42:02.596588 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.596568    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mlpxh_30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd/routeoverride-cni/0.log"
Jul 30 04:42:02.614146 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.614122    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mlpxh_30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd/whereabouts-cni-bincopy/0.log"
Jul 30 04:42:02.632444 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.632418    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mlpxh_30d5a13d-4f2c-4f63-8a1f-b85b3b7c0ccd/whereabouts-cni/0.log"
Jul 30 04:42:02.765167 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.765128    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pc47f_e5bd1a5c-9bd3-4874-991c-339874cf2ad0/network-metrics-daemon/0.log"
Jul 30 04:42:02.778805 ip-10-0-137-125 kubenswrapper[2560]: I0730 04:42:02.778730    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pc47f_e5bd1a5c-9bd3-4874-991c-339874cf2ad0/kube-rbac-proxy/0.log"