Jul 28 17:58:53.696611 ip-10-0-139-201 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 17:58:53.696732 ip-10-0-139-201 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 17:58:53.696793 ip-10-0-139-201 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 17:58:53.697238 ip-10-0-139-201 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 17:59:03.726560 ip-10-0-139-201 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 17:59:03.726583 ip-10-0-139-201 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 52bb51fce2c644328a734214f9286698 -- Jul 28 18:00:55.264921 ip-10-0-139-201 systemd[1]: Starting Kubernetes Kubelet... Jul 28 18:00:55.693053 ip-10-0-139-201 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 18:00:55.693053 ip-10-0-139-201 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 18:00:55.693053 ip-10-0-139-201 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 18:00:55.693053 ip-10-0-139-201 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 18:00:55.693053 ip-10-0-139-201 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 18:00:55.694941 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.694826 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 18:00:55.699145 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699129 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 18:00:55.699145 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699145 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699149 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699153 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699156 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699159 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699162 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699165 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699169 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699172 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699175 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699178 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699180 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699183 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699186 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699189 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699191 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699194 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699196 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699199 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699202 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 18:00:55.699210 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699204 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699207 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699209 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699212 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699215 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699218 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699220 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699224 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699238 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699241 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699244 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699246 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699249 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699252 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699254 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699257 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699259 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699262 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699264 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699269 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 18:00:55.699822 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699272 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699275 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699278 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699281 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699284 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699286 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699289 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699291 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699294 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699296 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699299 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699301 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699304 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699306 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699310 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699313 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699316 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699318 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699321 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 18:00:55.700330 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699324 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699326 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699329 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699334 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699338 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699341 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699344 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699346 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699349 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699352 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699355 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699358 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699361 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699363 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699366 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699368 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699371 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699374 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699376 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699380 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 18:00:55.700803 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699383 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699385 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699388 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699390 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699393 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699396 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699399 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699403 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699406 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699408 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699411 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699413 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699416 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699419 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699421 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699423 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699426 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699429 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 18:00:55.701305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.699433 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 18:00:55.701776 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701763 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 18:00:55.701776 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701775 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701779 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701783 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701786 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701789 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701793 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701796 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701799 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701801 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701804 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701807 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701809 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701812 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701814 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701817 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701819 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701822 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701824 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701827 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701830 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 18:00:55.701828 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701832 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701836 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701839 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701842 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701844 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701847 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701849 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701852 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701855 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701859 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701862 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701865 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701869 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701871 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701874 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701877 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701879 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701885 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701888 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701890 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 18:00:55.702343 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701893 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701895 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701898 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701900 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701903 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701905 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701908 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701910 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701913 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701916 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701918 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701921 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701923 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701925 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701928 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701932 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701934 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701937 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701939 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701942 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 18:00:55.702842 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701944 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701947 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701949 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701952 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701954 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701956 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701959 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701961 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701964 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701966 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701969 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701972 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701974 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701977 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701979 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701982 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701985 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701988 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701991 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701993 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 18:00:55.703346 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701996 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.701998 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702001 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702003 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702006 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702009 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702011 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702014 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702018 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702022 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702025 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702027 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702030 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702032 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702036 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702039 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702042 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702045 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702424 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 18:00:55.703851 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702429 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702432 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702435 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702438 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702441 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702444 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702447 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702449 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702452 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702455 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702458 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702460 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702463 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702467 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702471 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702474 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702477 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702480 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702482 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 18:00:55.704337 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702486 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702489 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702492 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702495 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702498 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702500 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702503 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702505 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702508 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702510 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702513 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702515 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702518 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702520 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702523 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702525 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702528 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702532 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702534 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702536 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 18:00:55.704795 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702539 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702541 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702544 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702548 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702552 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702555 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702558 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702561 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702564 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702568 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702570 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702573 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702576 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702579 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702582 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702585 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702588 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702591 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702593 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 18:00:55.705324 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702596 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702598 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702601 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702603 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702606 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702608 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702611 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702613 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702615 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702618 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702620 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702623 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702626 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702629 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702632 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702634 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702637 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702640 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702643 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702646 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 18:00:55.705783 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702648 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702651 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702654 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702662 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702666 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702669 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702671 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702674 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702676 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702679 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702681 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702684 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702686 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702689 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702691 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702694 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702696 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702698 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702701 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.702703 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 18:00:55.706319 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702773 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702786 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702792 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702797 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702801 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702805 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702808 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702812 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702816 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702820 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702823 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702827 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702830 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702833 2569 flags.go:64] FLAG: --cgroup-root="" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702836 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702839 2569 flags.go:64] FLAG: --client-ca-file="" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702842 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702845 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702850 2569 flags.go:64] FLAG: --cluster-domain="" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702853 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702856 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702859 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702863 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702866 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 18:00:55.706799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702869 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702872 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702875 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702878 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702881 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702884 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702887 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702890 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702894 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702897 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702900 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702903 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702906 2569 flags.go:64] FLAG: --enable-server="true" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702909 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702914 2569 flags.go:64] FLAG: --event-burst="100" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702917 2569 flags.go:64] FLAG: --event-qps="50" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702920 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702923 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702927 2569 flags.go:64] FLAG: --eviction-hard="" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702930 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702933 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702936 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702938 2569 flags.go:64] FLAG: --eviction-soft="" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702941 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702944 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 18:00:55.707381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702947 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702951 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702953 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702956 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702959 2569 flags.go:64] FLAG: --feature-gates="" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702962 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702965 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702968 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702971 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702975 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702978 2569 flags.go:64] FLAG: --help="false" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702981 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-139-201.ec2.internal" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702984 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702987 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702990 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702993 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702996 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.702999 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703002 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703005 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703007 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703011 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703014 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703017 2569 flags.go:64] FLAG: --kube-reserved="" Jul 28 18:00:55.708009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703022 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703025 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703028 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703031 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703034 2569 flags.go:64] FLAG: --lock-file="" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703037 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703039 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703042 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703047 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703050 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703053 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703056 2569 flags.go:64] FLAG: --logging-format="text" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703059 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703062 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703065 2569 flags.go:64] FLAG: --manifest-url="" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703070 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703074 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703077 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703080 2569 flags.go:64] FLAG: --max-pods="110" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703084 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703086 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703089 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703092 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703095 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703098 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 18:00:55.708614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703101 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703107 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703110 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703113 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703117 2569 flags.go:64] FLAG: --pod-cidr="" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703120 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703125 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703128 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703133 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703136 2569 flags.go:64] FLAG: --port="10250" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703139 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703142 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027b6386ba90bad48" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703145 2569 flags.go:64] FLAG: --qos-reserved="" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703148 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703151 2569 flags.go:64] FLAG: --register-node="true" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703154 2569 flags.go:64] FLAG: --register-with-taints="" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703157 2569 flags.go:64] FLAG: --registry-burst="10" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703160 2569 flags.go:64] FLAG: --registry-qps="5" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703163 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703166 2569 flags.go:64] FLAG: --reserved-memory="" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703169 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703172 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703177 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703180 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703183 2569 flags.go:64] FLAG: --runonce="false" Jul 28 18:00:55.709221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703185 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703189 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703191 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703194 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703197 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703200 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703203 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703206 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703209 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703212 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703215 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703218 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703221 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703224 2569 flags.go:64] FLAG: --system-cgroups="" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703240 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703246 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703250 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703253 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703257 2569 flags.go:64] FLAG: --tls-min-version="" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703260 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703262 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703266 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703269 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703272 2569 flags.go:64] FLAG: --v="2" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703276 2569 flags.go:64] FLAG: --version="false" Jul 28 18:00:55.710140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703280 2569 flags.go:64] FLAG: --vmodule="" Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703283 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.703286 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703415 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703419 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703423 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703426 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703429 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703432 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703434 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703437 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703439 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703442 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703445 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703447 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703450 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703452 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703456 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703460 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 18:00:55.711209 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703463 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703466 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703470 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703474 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703477 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703479 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703482 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703486 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703489 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703492 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703495 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703498 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703500 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703503 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703505 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703508 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703511 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703513 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703516 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703519 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 18:00:55.712046 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703522 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703524 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703526 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703529 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703532 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703534 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703537 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703539 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703542 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703544 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703547 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703550 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703552 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703555 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703557 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703562 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703565 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703567 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703570 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 18:00:55.712937 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703573 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703576 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703579 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703581 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703583 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703586 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703588 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703591 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703594 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703596 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703599 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703601 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703604 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703606 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703609 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703611 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703614 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703616 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703619 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703621 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 18:00:55.713535 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703624 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703626 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703629 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703631 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703634 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703636 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703639 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703641 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703645 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703648 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703651 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703653 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703656 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703658 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703661 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703663 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703666 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703668 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703671 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703673 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 18:00:55.714318 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703676 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703678 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703681 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.703685 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.704224 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.711716 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.711736 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.711759 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.711768 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711851 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711860 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711865 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711870 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711877 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 18:00:55.715131 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711884 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711890 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711895 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711900 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711905 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711910 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711914 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711919 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711923 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711928 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711933 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711938 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711942 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711947 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711951 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711956 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711960 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711964 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711968 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711974 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 18:00:55.715672 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711978 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711982 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711987 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711991 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.711997 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712001 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712005 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712010 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712016 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712022 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712027 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712031 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712036 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712040 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712044 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712048 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712052 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712056 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712060 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 18:00:55.716165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712064 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712068 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712073 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712077 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712082 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712086 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712090 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712095 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712099 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712103 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712108 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712111 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712116 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712121 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712126 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712133 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712137 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712143 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712148 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712153 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 18:00:55.716651 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712157 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712161 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712165 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712170 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712174 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712179 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712183 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712187 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712191 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712195 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712199 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712204 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712208 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712213 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712217 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712221 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712247 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712252 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712255 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712259 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 18:00:55.717459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712264 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712267 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712271 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712276 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712280 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712285 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712289 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712294 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712298 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712303 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712307 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712312 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712317 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712321 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712325 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 18:00:55.717978 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.712334 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712523 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712531 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712535 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712540 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712544 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712548 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712552 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712557 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712561 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712565 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712569 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712573 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712577 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712581 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712586 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712590 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712594 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712599 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712603 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 18:00:55.718383 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712606 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712611 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712615 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712620 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712624 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712628 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712632 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712636 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712640 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712646 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712651 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712655 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712659 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712663 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712667 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712671 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712675 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712679 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712683 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712687 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 18:00:55.718904 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712691 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712695 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712700 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712704 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712708 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712712 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712716 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712720 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712724 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712728 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712732 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712736 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712740 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712745 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712749 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712753 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712757 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712761 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712765 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712770 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 18:00:55.719416 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712774 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712779 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712783 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712789 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712795 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712799 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712803 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712807 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712811 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712815 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712819 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712823 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712827 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712831 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712835 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712840 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712844 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712848 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712852 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712856 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 18:00:55.719899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712860 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712865 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712871 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712876 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712881 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712886 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712890 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712895 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712899 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712905 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712911 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712916 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712920 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712924 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712929 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712934 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712939 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712943 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712948 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 18:00:55.720538 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:55.712952 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 18:00:55.721000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.712959 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 18:00:55.721000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.713105 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 18:00:55.721000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.717528 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 18:00:55.721000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.718966 2569 server.go:1019] "Starting client certificate rotation" Jul 28 18:00:55.721000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.719062 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 18:00:55.721000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.719090 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 18:00:55.722008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.721992 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 18:00:55.723136 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.723120 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 18:00:55.734693 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.734674 2569 log.go:25] "Validated CRI v1 runtime API" Jul 28 18:00:55.737276 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.737261 2569 log.go:25] "Validated CRI v1 image API" Jul 28 18:00:55.738056 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.738040 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 18:00:55.740404 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.740387 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c7d21cba-0189-4639-9f27-ed2933fb489d:/dev/nvme0n1p4 ca02c2da-a4cb-4dc7-87b9-1852dfb4d220:/dev/nvme0n1p3] Jul 28 18:00:55.740452 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.740404 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 18:00:55.746732 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.746621 2569 manager.go:216] Machine: {Timestamp:2026-07-28 18:00:55.745094538 +0000 UTC m=+0.358378939 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099206 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec204383c68b11e43bcd31fab8df80cd SystemUUID:ec204383-c68b-11e4-3bcd-31fab8df80cd BootID:52bb51fc-e2c6-4432-8a73-4214f9286698 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a4:2d:9e:b1:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a4:2d:9e:b1:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:72:7a:8c:6a:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 18:00:55.746732 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.746731 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 18:00:55.746860 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.746848 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 18:00:55.748571 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.748550 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 18:00:55.748737 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.748581 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 18:00:55.748820 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.748806 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 18:00:55.749890 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.749879 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 18:00:55.749921 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.749894 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 18:00:55.749921 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.749902 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 18:00:55.749982 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.749923 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 18:00:55.750773 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.750763 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 18:00:55.753595 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.753584 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 28 18:00:55.753720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.753711 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 18:00:55.753747 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.753743 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 28 18:00:55.753774 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.753752 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 18:00:55.753774 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.753769 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 18:00:55.753827 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.753777 2569 kubelet.go:398] "Adding apiserver pod source" Jul 28 18:00:55.753827 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.753786 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 18:00:55.754285 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.754266 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 18:00:55.754410 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.754306 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 18:00:55.757120 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.757090 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 18:00:55.759128 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.759111 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 18:00:55.759204 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.759143 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 18:00:55.760726 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760714 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 18:00:55.760773 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760730 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 18:00:55.760773 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760736 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 18:00:55.760773 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760742 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 18:00:55.760773 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760750 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 18:00:55.760773 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760756 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 18:00:55.760773 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760768 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 18:00:55.760924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760777 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 18:00:55.760924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760786 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 18:00:55.760924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760792 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 18:00:55.760924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760801 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 18:00:55.760924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.760806 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 18:00:55.761443 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.761432 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 18:00:55.761489 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.761444 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 18:00:55.764614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.764600 2569 server.go:1268] "Started kubelet" Jul 28 18:00:55.764710 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.764684 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 18:00:55.764748 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.764686 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 18:00:55.764784 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.764763 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 18:00:55.765479 ip-10-0-139-201 systemd[1]: Started Kubernetes Kubelet. Jul 28 18:00:55.766509 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.766481 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 18:00:55.767291 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.767276 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 28 18:00:55.767961 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.767938 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 18:00:55.768070 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.767986 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 18:00:55.768070 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.768003 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 18:00:55.768178 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.768134 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-89x7w" Jul 28 18:00:55.769880 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.769842 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 18:00:55.773089 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773075 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 18:00:55.773166 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773095 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 18:00:55.773166 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.773154 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:55.773558 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.772335 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-201.ec2.internal.18c6860466f91da6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-201.ec2.internal,UID:ip-10-0-139-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-201.ec2.internal,},FirstTimestamp:2026-07-28 18:00:55.764573606 +0000 UTC m=+0.377858006,LastTimestamp:2026-07-28 18:00:55.764573606 +0000 UTC m=+0.377858006,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-201.ec2.internal,}" Jul 28 18:00:55.773671 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773644 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 18:00:55.773671 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773660 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 18:00:55.773807 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773797 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 28 18:00:55.773865 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773809 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 28 18:00:55.773901 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773883 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 18:00:55.773986 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773971 2569 factory.go:55] Registering systemd factory Jul 28 18:00:55.774044 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.773990 2569 factory.go:223] Registration of the systemd container factory successfully Jul 28 18:00:55.774213 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.774200 2569 factory.go:153] Registering CRI-O factory Jul 28 18:00:55.774308 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.774215 2569 factory.go:223] Registration of the crio container factory successfully Jul 28 18:00:55.774308 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.774293 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 18:00:55.774413 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.774311 2569 factory.go:103] Registering Raw factory Jul 28 18:00:55.774413 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.774329 2569 manager.go:1196] Started watching for new ooms in manager Jul 28 18:00:55.775066 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.775052 2569 manager.go:349] Starting recovery of all containers Jul 28 18:00:55.776371 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.776349 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-89x7w" Jul 28 18:00:55.783711 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.783683 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 18:00:55.783816 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.783716 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 18:00:55.787806 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.787789 2569 manager.go:354] Recovery completed Jul 28 18:00:55.792332 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.792320 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 18:00:55.794190 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.794173 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Jul 28 18:00:55.794278 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.794210 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 18:00:55.794278 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.794239 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Jul 28 18:00:55.794793 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.794775 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 18:00:55.794793 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.794791 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 18:00:55.794902 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.794792 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 18:00:55.794902 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.794886 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 28 18:00:55.795497 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.795483 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Jul 28 18:00:55.795545 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.795511 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 18:00:55.795545 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.795524 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Jul 28 18:00:55.796702 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.796629 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-201.ec2.internal.18c6860468bd117c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-201.ec2.internal,UID:ip-10-0-139-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-201.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-201.ec2.internal,},FirstTimestamp:2026-07-28 18:00:55.794192764 +0000 UTC m=+0.407477166,LastTimestamp:2026-07-28 18:00:55.794192764 +0000 UTC m=+0.407477166,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-201.ec2.internal,}" Jul 28 18:00:55.797665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.797653 2569 policy_none.go:49] "None policy: Start" Jul 28 18:00:55.797722 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.797672 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 18:00:55.797722 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.797686 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 18:00:55.799705 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.799692 2569 policy_none.go:47] "Start" Jul 28 18:00:55.838994 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.838798 2569 manager.go:337] "Starting Device Plugin manager" Jul 28 18:00:55.839087 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.839032 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 18:00:55.839087 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.839043 2569 server.go:87] "Starting device plugin registration server" Jul 28 18:00:55.839298 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.839275 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 18:00:55.839409 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.839291 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 18:00:55.839409 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.839381 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 18:00:55.839527 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.839459 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 18:00:55.839527 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.839469 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 18:00:55.840041 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.840019 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 18:00:55.840107 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.840082 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:55.939974 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.939948 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 18:00:55.940922 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.940906 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Jul 28 18:00:55.941000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.940940 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 18:00:55.941000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.940956 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Jul 28 18:00:55.941000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.940991 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-201.ec2.internal" Jul 28 18:00:55.949203 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.949155 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-201.ec2.internal" Jul 28 18:00:55.949203 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.949182 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-201.ec2.internal\": node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:55.964305 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.964274 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:55.967943 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.967919 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 18:00:55.969118 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.969096 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 18:00:55.969175 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.969158 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 18:00:55.969211 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.969185 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 18:00:55.969254 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:55.969229 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 18:00:55.971255 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:55.971231 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 18:00:56.067687 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.067648 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.069989 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.069965 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"] Jul 28 18:00:56.070088 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.070075 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 18:00:56.071765 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.071747 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Jul 28 18:00:56.071855 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.071778 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 18:00:56.071855 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.071790 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Jul 28 18:00:56.074151 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074138 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 18:00:56.074251 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074226 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.074322 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074269 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 18:00:56.074381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c24b599316ee2807955aed8606f79acb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"c24b599316ee2807955aed8606f79acb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.074424 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c24b599316ee2807955aed8606f79acb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"c24b599316ee2807955aed8606f79acb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.074915 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074900 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Jul 28 18:00:56.074986 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074931 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 18:00:56.074986 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074942 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Jul 28 18:00:56.074986 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.074900 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Jul 28 18:00:56.075078 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.075004 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 18:00:56.075078 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.075016 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Jul 28 18:00:56.077210 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.077197 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.077269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.077225 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 18:00:56.077878 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.077861 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Jul 28 18:00:56.077967 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.077888 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 18:00:56.077967 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.077899 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Jul 28 18:00:56.100127 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.100111 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.104810 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.104794 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.168023 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.167994 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.175242 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.175202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c24b599316ee2807955aed8606f79acb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"c24b599316ee2807955aed8606f79acb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.175343 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.175257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c24b599316ee2807955aed8606f79acb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"c24b599316ee2807955aed8606f79acb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.175343 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.175300 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c24b599316ee2807955aed8606f79acb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"c24b599316ee2807955aed8606f79acb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.175343 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.175325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f02759f6c62b6d84c775f3fa1d163393-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"f02759f6c62b6d84c775f3fa1d163393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.175343 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.175300 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c24b599316ee2807955aed8606f79acb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"c24b599316ee2807955aed8606f79acb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.276990 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.269588 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.276990 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.275724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f02759f6c62b6d84c775f3fa1d163393-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"f02759f6c62b6d84c775f3fa1d163393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.276990 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.275761 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f02759f6c62b6d84c775f3fa1d163393-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"f02759f6c62b6d84c775f3fa1d163393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.377103 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.377068 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.405408 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.405374 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.408699 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.408682 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Jul 28 18:00:56.477307 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.477281 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.578111 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.578028 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.630994 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.630967 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 18:00:56.678885 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.678851 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.718019 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.717994 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 18:00:56.718448 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.718120 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 18:00:56.718448 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.718165 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 18:00:56.779931 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.779907 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.787954 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.787926 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:55:56 +0000 UTC" deadline="2028-02-04 22:17:36.311786135 +0000 UTC" Jul 28 18:00:56.787954 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:56.787954 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13348h16m39.52383641s" Jul 28 18:00:56.880980 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.880955 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:56.981337 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:56.981302 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:57.069038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.069007 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 18:00:57.081619 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.081585 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:57.182185 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.182122 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:57.283033 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.283003 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Jul 28 18:00:57.353664 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.353639 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 18:00:57.373899 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.373877 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Jul 28 18:00:57.385720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.385704 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 18:00:57.386883 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.386869 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Jul 28 18:00:57.393788 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.393771 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 18:00:57.499269 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:57.499223 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02759f6c62b6d84c775f3fa1d163393.slice/crio-6c4819c5391d17627fef17d4d3f47fe2b97bbc5041893389f88a6463d5e41131 WatchSource:0}: Error finding container 6c4819c5391d17627fef17d4d3f47fe2b97bbc5041893389f88a6463d5e41131: Status 404 returned error can't find the container with id 6c4819c5391d17627fef17d4d3f47fe2b97bbc5041893389f88a6463d5e41131 Jul 28 18:00:57.503547 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.503531 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:00:57.755440 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.755366 2569 apiserver.go:52] "Watching apiserver" Jul 28 18:00:57.764006 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.763988 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 18:00:57.764474 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.764452 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-d4cgv","kube-system/global-pull-secret-syncer-jw4r5","kube-system/konnectivity-agent-vn5cx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d","openshift-image-registry/node-ca-ds5hw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-8lxf9","kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal","openshift-cluster-node-tuning-operator/tuned-56lln","openshift-dns/node-resolver-w829b","openshift-multus/multus-additional-cni-plugins-84xx7","openshift-multus/multus-pvms7","openshift-multus/network-metrics-daemon-8z6t7","openshift-network-diagnostics/network-check-target-nwkjt"] Jul 28 18:00:57.767075 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.767057 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.769326 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.769299 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:57.769436 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.769411 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:00:57.769686 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.769666 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 18:00:57.769971 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.769951 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 18:00:57.770045 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.769973 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 18:00:57.770101 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.769958 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 18:00:57.770155 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.770135 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 18:00:57.771480 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.771461 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 18:00:57.771616 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.771601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:00:57.771703 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.771687 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p76jd\"" Jul 28 18:00:57.773527 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.773497 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 18:00:57.774094 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.774073 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.774517 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.774261 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gxjfd\"" Jul 28 18:00:57.774626 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.774603 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 18:00:57.775119 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.775074 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 18:00:57.777583 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.776550 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 18:00:57.777583 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.777048 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sv8m7\"" Jul 28 18:00:57.777583 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.777348 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 18:00:57.777583 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.777530 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 18:00:57.778178 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.778159 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.780954 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.780936 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7psn4\"" Jul 28 18:00:57.781062 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.780948 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 18:00:57.781197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.781181 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 18:00:57.781266 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.781215 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 18:00:57.781360 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.781303 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:00:57.781429 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.781378 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:00:57.783838 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.783818 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.784935 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.784919 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 18:00:57.785741 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-device-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.785835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-kubelet\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.785835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-cni-netd\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.785835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:57.785835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-slash\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785853 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-node-log\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-run-netns\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785901 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-ovn\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785932 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62704919-bf62-4605-b9ee-8749f46bb72d-konnectivity-ca\") pod \"konnectivity-agent-vn5cx\" (UID: \"62704919-bf62-4605-b9ee-8749f46bb72d\") " pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:00:57.786008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785968 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.786008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.785991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-sys-fs\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.786231 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-systemd\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786231 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-var-lib-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786231 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-ovnkube-script-lib\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786231 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dmf\" (UniqueName: \"kubernetes.io/projected/8b61e46a-8023-4339-b302-2608710d3964-kube-api-access-64dmf\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786231 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786191 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-registration-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w575v\" (UniqueName: \"kubernetes.io/projected/2710a9c1-e24a-41ad-be7a-c87d246c82ad-kube-api-access-w575v\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786341 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-ovnkube-config\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786349 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b61e46a-8023-4339-b302-2608710d3964-ovn-node-metrics-cert\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786396 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62704919-bf62-4605-b9ee-8749f46bb72d-agent-certs\") pod \"konnectivity-agent-vn5cx\" (UID: \"62704919-bf62-4605-b9ee-8749f46bb72d\") " pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-socket-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786469 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-log-socket\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786493 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-cni-bin\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786509 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbxrc\"" Jul 28 18:00:57.786518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.787027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786543 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-env-overrides\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.787027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786572 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.787027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786597 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.787027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786604 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-systemd-units\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.787027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-etc-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.787027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.787027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786705 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cp25\" (UniqueName: \"kubernetes.io/projected/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-kube-api-access-8cp25\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:57.787027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.786516 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 18:00:57.787662 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:57.787629 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24b599316ee2807955aed8606f79acb.slice/crio-2d03dd2172f8e9f191fbe50a09bd85126026a31fbe3888ef8cef645b90acd9ac WatchSource:0}: Error finding container 2d03dd2172f8e9f191fbe50a09bd85126026a31fbe3888ef8cef645b90acd9ac: Status 404 returned error can't find the container with id 2d03dd2172f8e9f191fbe50a09bd85126026a31fbe3888ef8cef645b90acd9ac Jul 28 18:00:57.789180 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.789119 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.789251 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.789182 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:00:57.790182 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.790011 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5pbqg\"" Jul 28 18:00:57.790182 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.790020 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 18:00:57.790182 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.790059 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 18:00:57.790182 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.790088 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 18:00:57.791941 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.791857 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.794520 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.794498 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.794803 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.794674 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d68mp\"" Jul 28 18:00:57.794803 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.794685 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 18:00:57.794803 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.794751 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 18:00:57.796959 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.796938 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.797198 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.797182 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 18:00:57.798148 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.798131 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 18:00:57.798339 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.798324 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 18:00:57.798562 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.798545 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cskxq\"" Jul 28 18:00:57.798712 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.798694 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 18:00:57.798782 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.798756 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 18:00:57.799976 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.799956 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gjvvz\"" Jul 28 18:00:57.800080 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.799989 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 18:00:57.826844 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.826827 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rxm5c" Jul 28 18:00:57.827399 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.827382 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 18:00:57.836180 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.836163 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rxm5c" Jul 28 18:00:57.874741 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.874717 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 18:00:57.886851 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.886826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62704919-bf62-4605-b9ee-8749f46bb72d-konnectivity-ca\") pod \"konnectivity-agent-vn5cx\" (UID: \"62704919-bf62-4605-b9ee-8749f46bb72d\") " pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:00:57.886931 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.886862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-sys-fs\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.886931 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.886881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-var-lib-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.886931 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.886898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-ovnkube-script-lib\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.886931 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.886916 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-host\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.886967 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-var-lib-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.886982 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-sys-fs\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-cni-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887027 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/057dfc76-9edb-4849-9f5c-f33869d37b32-cni-binary-copy\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-kubelet\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887059 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-systemd\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887091 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-conf-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-cni-bin\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887149 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887174 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-run\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887247 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887225 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-var-lib-kubelet\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887276 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-cni-bin\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-system-cni-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887312 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb20a3ca-9707-403d-a6d1-6093065b2a46-host\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887334 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-systemd-units\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887372 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887387 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-systemd-units\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887423 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-multus-certs\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887431 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-etc-kubernetes\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.887720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-device-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-kubelet\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887546 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-system-cni-dir\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887551 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-device-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887578 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-ovnkube-script-lib\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887599 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62704919-bf62-4605-b9ee-8749f46bb72d-konnectivity-ca\") pod \"konnectivity-agent-vn5cx\" (UID: \"62704919-bf62-4605-b9ee-8749f46bb72d\") " pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887578 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-kubelet\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887575 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-os-release\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-hostroot\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887687 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-node-log\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887712 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/355dd9c8-babd-4952-b249-a5b5c5df07e6-iptables-alerter-script\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887740 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-node-log\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887738 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysctl-conf\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-socket-dir-parent\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887790 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-run-netns\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb20a3ca-9707-403d-a6d1-6093065b2a46-serviceca\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887831 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-run-netns\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.888347 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887849 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d42d3962-0660-48ee-8248-66d18ead7206-tmp-dir\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-systemd\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887912 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dmf\" (UniqueName: \"kubernetes.io/projected/8b61e46a-8023-4339-b302-2608710d3964-kube-api-access-64dmf\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887921 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887934 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-systemd\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.887968 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fd18dc0-806e-46c1-811c-31680ca4fbdb-kubelet-config\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9f94\" (UniqueName: \"kubernetes.io/projected/057dfc76-9edb-4849-9f5c-f33869d37b32-kube-api-access-k9f94\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-registration-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w575v\" (UniqueName: \"kubernetes.io/projected/2710a9c1-e24a-41ad-be7a-c87d246c82ad-kube-api-access-w575v\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-ovnkube-config\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888190 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-registration-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b61e46a-8023-4339-b302-2608710d3964-ovn-node-metrics-cert\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888336 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysctl-d\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888353 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34de2e65-d720-4bdc-b886-978d18c29e1c-tmp\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwh8\" (UniqueName: \"kubernetes.io/projected/34de2e65-d720-4bdc-b886-978d18c29e1c-kube-api-access-ldwh8\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888397 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-daemon-config\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.889057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888416 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62704919-bf62-4605-b9ee-8749f46bb72d-agent-certs\") pod \"konnectivity-agent-vn5cx\" (UID: \"62704919-bf62-4605-b9ee-8749f46bb72d\") " pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-socket-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-log-socket\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-env-overrides\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888564 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-socket-dir\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/355dd9c8-babd-4952-b249-a5b5c5df07e6-host-slash\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888609 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-log-socket\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888619 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4tr\" (UniqueName: \"kubernetes.io/projected/d42d3962-0660-48ee-8248-66d18ead7206-kube-api-access-dw4tr\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-modprobe-d\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888684 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-sys\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-lib-modules\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888727 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-ovnkube-config\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d42d3962-0660-48ee-8248-66d18ead7206-hosts-file\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-cni-bin\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888843 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-etc-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.889688 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fd18dc0-806e-46c1-811c-31680ca4fbdb-dbus\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-tuned\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888906 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b61e46a-8023-4339-b302-2608710d3964-env-overrides\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cp25\" (UniqueName: \"kubernetes.io/projected/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-kube-api-access-8cp25\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888914 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-etc-openvswitch\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-cni-netd\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888964 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7jhq\" (UniqueName: \"kubernetes.io/projected/355dd9c8-babd-4952-b249-a5b5c5df07e6-kube-api-access-p7jhq\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888974 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2710a9c1-e24a-41ad-be7a-c87d246c82ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-kubernetes\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.888999 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-cni-netd\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cnibin\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-os-release\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889061 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889098 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-k8s-cni-cncf-io\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-slash\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.890197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889179 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hkw\" (UniqueName: \"kubernetes.io/projected/bb20a3ca-9707-403d-a6d1-6093065b2a46-kube-api-access-65hkw\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889215 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-cnibin\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.889218 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889251 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-cni-multus\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889281 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-host-slash\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.889334 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs podName:c27d9f56-189f-4f1d-bd3b-60fb56978bb3 nodeName:}" failed. No retries permitted until 2026-07-28 18:00:58.389269564 +0000 UTC m=+3.002553972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs") pod "network-metrics-daemon-8z6t7" (UID: "c27d9f56-189f-4f1d-bd3b-60fb56978bb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889378 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-ovn\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889356 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b61e46a-8023-4339-b302-2608710d3964-run-ovn\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889431 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysconfig\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889454 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h75z\" (UniqueName: \"kubernetes.io/projected/43195a1e-92f0-4c69-bef3-c529d4c75fd8-kube-api-access-7h75z\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.890690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.889475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-netns\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.891946 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.891927 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b61e46a-8023-4339-b302-2608710d3964-ovn-node-metrics-cert\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.892144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.892128 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62704919-bf62-4605-b9ee-8749f46bb72d-agent-certs\") pod \"konnectivity-agent-vn5cx\" (UID: \"62704919-bf62-4605-b9ee-8749f46bb72d\") " pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:00:57.901317 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.901282 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w575v\" (UniqueName: \"kubernetes.io/projected/2710a9c1-e24a-41ad-be7a-c87d246c82ad-kube-api-access-w575v\") pod \"aws-ebs-csi-driver-node-lq97d\" (UID: \"2710a9c1-e24a-41ad-be7a-c87d246c82ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:57.901539 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.901523 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dmf\" (UniqueName: \"kubernetes.io/projected/8b61e46a-8023-4339-b302-2608710d3964-kube-api-access-64dmf\") pod \"ovnkube-node-8lxf9\" (UID: \"8b61e46a-8023-4339-b302-2608710d3964\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:57.901583 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.901560 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cp25\" (UniqueName: \"kubernetes.io/projected/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-kube-api-access-8cp25\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:57.978510 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.978463 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" event={"ID":"f02759f6c62b6d84c775f3fa1d163393","Type":"ContainerStarted","Data":"6c4819c5391d17627fef17d4d3f47fe2b97bbc5041893389f88a6463d5e41131"} Jul 28 18:00:57.979650 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.979622 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"c24b599316ee2807955aed8606f79acb","Type":"ContainerStarted","Data":"2d03dd2172f8e9f191fbe50a09bd85126026a31fbe3888ef8cef645b90acd9ac"} Jul 28 18:00:57.990531 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990510 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-host\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.990670 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-cni-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.990670 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/057dfc76-9edb-4849-9f5c-f33869d37b32-cni-binary-copy\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.990670 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990632 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-host\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-kubelet\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990734 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-systemd\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990757 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-cni-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-conf-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990786 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-kubelet\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990812 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990824 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-conf-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.990831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-run\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.990845 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990890 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-var-lib-kubelet\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990894 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-run\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-systemd\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-var-lib-kubelet\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.990938 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret podName:8fd18dc0-806e-46c1-811c-31680ca4fbdb nodeName:}" failed. No retries permitted until 2026-07-28 18:00:58.490923931 +0000 UTC m=+3.104208321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret") pod "global-pull-secret-syncer-jw4r5" (UID: "8fd18dc0-806e-46c1-811c-31680ca4fbdb") : object "kube-system"/"original-pull-secret" not registered Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990955 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-system-cni-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.990984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb20a3ca-9707-403d-a6d1-6093065b2a46-host\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991043 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-system-cni-dir\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-multus-certs\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991046 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb20a3ca-9707-403d-a6d1-6093065b2a46-host\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-etc-kubernetes\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991124 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-etc-kubernetes\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-system-cni-dir\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.991264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991160 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-multus-certs\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-os-release\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991174 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991180 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-system-cni-dir\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-hostroot\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991222 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-hostroot\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991260 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-os-release\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991251 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/057dfc76-9edb-4849-9f5c-f33869d37b32-cni-binary-copy\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/355dd9c8-babd-4952-b249-a5b5c5df07e6-iptables-alerter-script\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysctl-conf\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-socket-dir-parent\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb20a3ca-9707-403d-a6d1-6093065b2a46-serviceca\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991375 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d42d3962-0660-48ee-8248-66d18ead7206-tmp-dir\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991374 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-socket-dir-parent\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fd18dc0-806e-46c1-811c-31680ca4fbdb-kubelet-config\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991414 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysctl-conf\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991431 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9f94\" (UniqueName: \"kubernetes.io/projected/057dfc76-9edb-4849-9f5c-f33869d37b32-kube-api-access-k9f94\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysctl-d\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.992065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991508 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fd18dc0-806e-46c1-811c-31680ca4fbdb-kubelet-config\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991594 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysctl-d\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34de2e65-d720-4bdc-b886-978d18c29e1c-tmp\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991645 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwh8\" (UniqueName: \"kubernetes.io/projected/34de2e65-d720-4bdc-b886-978d18c29e1c-kube-api-access-ldwh8\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-daemon-config\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991710 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/355dd9c8-babd-4952-b249-a5b5c5df07e6-host-slash\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991714 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991737 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4tr\" (UniqueName: \"kubernetes.io/projected/d42d3962-0660-48ee-8248-66d18ead7206-kube-api-access-dw4tr\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991780 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-modprobe-d\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-sys\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991820 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/355dd9c8-babd-4952-b249-a5b5c5df07e6-iptables-alerter-script\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-lib-modules\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991840 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb20a3ca-9707-403d-a6d1-6093065b2a46-serviceca\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991719 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d42d3962-0660-48ee-8248-66d18ead7206-tmp-dir\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d42d3962-0660-48ee-8248-66d18ead7206-hosts-file\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991907 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-sys\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991918 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d42d3962-0660-48ee-8248-66d18ead7206-hosts-file\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991930 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-modprobe-d\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991935 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-cni-bin\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991952 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/355dd9c8-babd-4952-b249-a5b5c5df07e6-host-slash\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991970 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fd18dc0-806e-46c1-811c-31680ca4fbdb-dbus\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.991996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-tuned\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992003 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-cni-bin\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992025 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7jhq\" (UniqueName: \"kubernetes.io/projected/355dd9c8-babd-4952-b249-a5b5c5df07e6-kube-api-access-p7jhq\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992044 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-lib-modules\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992050 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fd18dc0-806e-46c1-811c-31680ca4fbdb-dbus\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992053 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-kubernetes\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cnibin\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-os-release\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992130 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cnibin\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992136 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-kubernetes\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992176 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43195a1e-92f0-4c69-bef3-c529d4c75fd8-os-release\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992185 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/057dfc76-9edb-4849-9f5c-f33869d37b32-multus-daemon-config\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-k8s-cni-cncf-io\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.993636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hkw\" (UniqueName: \"kubernetes.io/projected/bb20a3ca-9707-403d-a6d1-6093065b2a46-kube-api-access-65hkw\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992336 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-cnibin\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992361 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-k8s-cni-cncf-io\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992386 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-cni-multus\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992590 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-cnibin\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992635 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-var-lib-cni-multus\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992644 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992663 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysconfig\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h75z\" (UniqueName: \"kubernetes.io/projected/43195a1e-92f0-4c69-bef3-c529d4c75fd8-kube-api-access-7h75z\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992725 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-netns\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992727 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-sysconfig\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.992884 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/057dfc76-9edb-4849-9f5c-f33869d37b32-host-run-netns\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:57.994134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.993536 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43195a1e-92f0-4c69-bef3-c529d4c75fd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:57.994625 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.994390 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34de2e65-d720-4bdc-b886-978d18c29e1c-etc-tuned\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.994625 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:57.994542 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34de2e65-d720-4bdc-b886-978d18c29e1c-tmp\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:57.999742 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.999722 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 18:00:57.999742 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.999745 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 18:00:57.999917 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.999756 2569 projected.go:196] Error preparing data for projected volume kube-api-access-p2zrd for pod openshift-network-diagnostics/network-check-target-nwkjt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:00:57.999917 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:57.999805 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd podName:37c30d72-01bf-4c69-a023-cccaa157ef96 nodeName:}" failed. No retries permitted until 2026-07-28 18:00:58.499793269 +0000 UTC m=+3.113077659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p2zrd" (UniqueName: "kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd") pod "network-check-target-nwkjt" (UID: "37c30d72-01bf-4c69-a023-cccaa157ef96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:00:58.002617 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.002584 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h75z\" (UniqueName: \"kubernetes.io/projected/43195a1e-92f0-4c69-bef3-c529d4c75fd8-kube-api-access-7h75z\") pod \"multus-additional-cni-plugins-84xx7\" (UID: \"43195a1e-92f0-4c69-bef3-c529d4c75fd8\") " pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:58.002830 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.002811 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9f94\" (UniqueName: \"kubernetes.io/projected/057dfc76-9edb-4849-9f5c-f33869d37b32-kube-api-access-k9f94\") pod \"multus-pvms7\" (UID: \"057dfc76-9edb-4849-9f5c-f33869d37b32\") " pod="openshift-multus/multus-pvms7" Jul 28 18:00:58.002970 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.002952 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4tr\" (UniqueName: \"kubernetes.io/projected/d42d3962-0660-48ee-8248-66d18ead7206-kube-api-access-dw4tr\") pod \"node-resolver-w829b\" (UID: \"d42d3962-0660-48ee-8248-66d18ead7206\") " pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:58.003184 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.003167 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7jhq\" (UniqueName: \"kubernetes.io/projected/355dd9c8-babd-4952-b249-a5b5c5df07e6-kube-api-access-p7jhq\") pod \"iptables-alerter-d4cgv\" (UID: \"355dd9c8-babd-4952-b249-a5b5c5df07e6\") " pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:58.003379 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.003362 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwh8\" (UniqueName: \"kubernetes.io/projected/34de2e65-d720-4bdc-b886-978d18c29e1c-kube-api-access-ldwh8\") pod \"tuned-56lln\" (UID: \"34de2e65-d720-4bdc-b886-978d18c29e1c\") " pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:58.003633 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.003618 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hkw\" (UniqueName: \"kubernetes.io/projected/bb20a3ca-9707-403d-a6d1-6093065b2a46-kube-api-access-65hkw\") pod \"node-ca-ds5hw\" (UID: \"bb20a3ca-9707-403d-a6d1-6093065b2a46\") " pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:58.078349 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.078276 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:00:58.086113 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.086092 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:00:58.088867 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.088841 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b61e46a_8023_4339_b302_2608710d3964.slice/crio-4e5cc07f8b7174fe0278528df9534a2d289e24c33a383dd1cf2ff7d001dff0c2 WatchSource:0}: Error finding container 4e5cc07f8b7174fe0278528df9534a2d289e24c33a383dd1cf2ff7d001dff0c2: Status 404 returned error can't find the container with id 4e5cc07f8b7174fe0278528df9534a2d289e24c33a383dd1cf2ff7d001dff0c2 Jul 28 18:00:58.094084 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.094061 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" Jul 28 18:00:58.094777 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.094752 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62704919_bf62_4605_b9ee_8749f46bb72d.slice/crio-aa6dc90c6eed687553cabe03d6b7e034777966244fe4ea77a20dd43aaaff2ce6 WatchSource:0}: Error finding container aa6dc90c6eed687553cabe03d6b7e034777966244fe4ea77a20dd43aaaff2ce6: Status 404 returned error can't find the container with id aa6dc90c6eed687553cabe03d6b7e034777966244fe4ea77a20dd43aaaff2ce6 Jul 28 18:00:58.099017 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.098999 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ds5hw" Jul 28 18:00:58.105697 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.105675 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-56lln" Jul 28 18:00:58.107070 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.107044 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2710a9c1_e24a_41ad_be7a_c87d246c82ad.slice/crio-92362a94d351f1c62404bcdbf30f56a60e7b61ec66b4d1873b4782531775d446 WatchSource:0}: Error finding container 92362a94d351f1c62404bcdbf30f56a60e7b61ec66b4d1873b4782531775d446: Status 404 returned error can't find the container with id 92362a94d351f1c62404bcdbf30f56a60e7b61ec66b4d1873b4782531775d446 Jul 28 18:00:58.111237 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.111222 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d4cgv" Jul 28 18:00:58.111442 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.111415 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb20a3ca_9707_403d_a6d1_6093065b2a46.slice/crio-c47abc5eaef67c0745d589e588d3fb3aa786d77a16aaf2822ca1092913569a2f WatchSource:0}: Error finding container c47abc5eaef67c0745d589e588d3fb3aa786d77a16aaf2822ca1092913569a2f: Status 404 returned error can't find the container with id c47abc5eaef67c0745d589e588d3fb3aa786d77a16aaf2822ca1092913569a2f Jul 28 18:00:58.116307 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.116266 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34de2e65_d720_4bdc_b886_978d18c29e1c.slice/crio-37d52a2220dab765cedc17184691dfc8be4c72722c03a4bafdaa54efbc761a1b WatchSource:0}: Error finding container 37d52a2220dab765cedc17184691dfc8be4c72722c03a4bafdaa54efbc761a1b: Status 404 returned error can't find the container with id 37d52a2220dab765cedc17184691dfc8be4c72722c03a4bafdaa54efbc761a1b Jul 28 18:00:58.116909 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.116888 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w829b" Jul 28 18:00:58.121070 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.121054 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355dd9c8_babd_4952_b249_a5b5c5df07e6.slice/crio-8267c8d7c5f54d963539e9132d5302530ea6e502fc2bfe284bd160d4890d69c5 WatchSource:0}: Error finding container 8267c8d7c5f54d963539e9132d5302530ea6e502fc2bfe284bd160d4890d69c5: Status 404 returned error can't find the container with id 8267c8d7c5f54d963539e9132d5302530ea6e502fc2bfe284bd160d4890d69c5 Jul 28 18:00:58.123150 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.123128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84xx7" Jul 28 18:00:58.126826 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.126801 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42d3962_0660_48ee_8248_66d18ead7206.slice/crio-85a58975fed6e666182d3c8bcd316f0b512ffbe9de05e008738d33626b45bba3 WatchSource:0}: Error finding container 85a58975fed6e666182d3c8bcd316f0b512ffbe9de05e008738d33626b45bba3: Status 404 returned error can't find the container with id 85a58975fed6e666182d3c8bcd316f0b512ffbe9de05e008738d33626b45bba3 Jul 28 18:00:58.127583 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.127560 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvms7" Jul 28 18:00:58.132912 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.132892 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43195a1e_92f0_4c69_bef3_c529d4c75fd8.slice/crio-9081c6d5dc4cabc4be300992138b5eea945ac5493832d9931109be898d081edb WatchSource:0}: Error finding container 9081c6d5dc4cabc4be300992138b5eea945ac5493832d9931109be898d081edb: Status 404 returned error can't find the container with id 9081c6d5dc4cabc4be300992138b5eea945ac5493832d9931109be898d081edb Jul 28 18:00:58.138371 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:00:58.138352 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057dfc76_9edb_4849_9f5c_f33869d37b32.slice/crio-ba270f508660d81cabb540fa108ef8ba36a7008cac45c3e607ec352d8b9ebf6a WatchSource:0}: Error finding container ba270f508660d81cabb540fa108ef8ba36a7008cac45c3e607ec352d8b9ebf6a: Status 404 returned error can't find the container with id ba270f508660d81cabb540fa108ef8ba36a7008cac45c3e607ec352d8b9ebf6a Jul 28 18:00:58.395923 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.395883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:58.396106 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.396027 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:00:58.396171 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.396110 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs podName:c27d9f56-189f-4f1d-bd3b-60fb56978bb3 nodeName:}" failed. No retries permitted until 2026-07-28 18:00:59.396089607 +0000 UTC m=+4.009374001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs") pod "network-metrics-daemon-8z6t7" (UID: "c27d9f56-189f-4f1d-bd3b-60fb56978bb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:00:58.411762 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.411731 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 18:00:58.497816 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.497207 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:58.497816 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.497403 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 18:00:58.497816 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.497473 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret podName:8fd18dc0-806e-46c1-811c-31680ca4fbdb nodeName:}" failed. No retries permitted until 2026-07-28 18:00:59.497452508 +0000 UTC m=+4.110736903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret") pod "global-pull-secret-syncer-jw4r5" (UID: "8fd18dc0-806e-46c1-811c-31680ca4fbdb") : object "kube-system"/"original-pull-secret" not registered Jul 28 18:00:58.598199 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.598159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:00:58.598449 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.598427 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 18:00:58.598534 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.598461 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 18:00:58.598534 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.598474 2569 projected.go:196] Error preparing data for projected volume kube-api-access-p2zrd for pod openshift-network-diagnostics/network-check-target-nwkjt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:00:58.598663 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.598539 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd podName:37c30d72-01bf-4c69-a023-cccaa157ef96 nodeName:}" failed. No retries permitted until 2026-07-28 18:00:59.598520575 +0000 UTC m=+4.211805030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2zrd" (UniqueName: "kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd") pod "network-check-target-nwkjt" (UID: "37c30d72-01bf-4c69-a023-cccaa157ef96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:00:58.840836 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.840663 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:55:58 +0000 UTC" deadline="2028-04-09 22:23:58.68493586 +0000 UTC" Jul 28 18:00:58.840836 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.840722 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14908h22m59.844218561s" Jul 28 18:00:58.970608 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:58.970569 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:00:58.970787 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:58.970712 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:00:59.003226 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.003186 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvms7" event={"ID":"057dfc76-9edb-4849-9f5c-f33869d37b32","Type":"ContainerStarted","Data":"ba270f508660d81cabb540fa108ef8ba36a7008cac45c3e607ec352d8b9ebf6a"} Jul 28 18:00:59.014606 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.014554 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d4cgv" event={"ID":"355dd9c8-babd-4952-b249-a5b5c5df07e6","Type":"ContainerStarted","Data":"8267c8d7c5f54d963539e9132d5302530ea6e502fc2bfe284bd160d4890d69c5"} Jul 28 18:00:59.032599 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.032368 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ds5hw" event={"ID":"bb20a3ca-9707-403d-a6d1-6093065b2a46","Type":"ContainerStarted","Data":"c47abc5eaef67c0745d589e588d3fb3aa786d77a16aaf2822ca1092913569a2f"} Jul 28 18:00:59.061267 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.061226 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" event={"ID":"2710a9c1-e24a-41ad-be7a-c87d246c82ad","Type":"ContainerStarted","Data":"92362a94d351f1c62404bcdbf30f56a60e7b61ec66b4d1873b4782531775d446"} Jul 28 18:00:59.063283 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.063238 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"4e5cc07f8b7174fe0278528df9534a2d289e24c33a383dd1cf2ff7d001dff0c2"} Jul 28 18:00:59.076211 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.076138 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerStarted","Data":"9081c6d5dc4cabc4be300992138b5eea945ac5493832d9931109be898d081edb"} Jul 28 18:00:59.087884 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.087851 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w829b" event={"ID":"d42d3962-0660-48ee-8248-66d18ead7206","Type":"ContainerStarted","Data":"85a58975fed6e666182d3c8bcd316f0b512ffbe9de05e008738d33626b45bba3"} Jul 28 18:00:59.094452 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.094371 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-56lln" event={"ID":"34de2e65-d720-4bdc-b886-978d18c29e1c","Type":"ContainerStarted","Data":"37d52a2220dab765cedc17184691dfc8be4c72722c03a4bafdaa54efbc761a1b"} Jul 28 18:00:59.100298 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.100246 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vn5cx" event={"ID":"62704919-bf62-4605-b9ee-8749f46bb72d","Type":"ContainerStarted","Data":"aa6dc90c6eed687553cabe03d6b7e034777966244fe4ea77a20dd43aaaff2ce6"} Jul 28 18:00:59.408425 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.408331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:59.408601 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.408561 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:00:59.408668 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.408629 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs podName:c27d9f56-189f-4f1d-bd3b-60fb56978bb3 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:01.408610568 +0000 UTC m=+6.021894961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs") pod "network-metrics-daemon-8z6t7" (UID: "c27d9f56-189f-4f1d-bd3b-60fb56978bb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:00:59.509724 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.509686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:59.509897 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.509827 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 18:00:59.509897 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.509892 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret podName:8fd18dc0-806e-46c1-811c-31680ca4fbdb nodeName:}" failed. No retries permitted until 2026-07-28 18:01:01.509872815 +0000 UTC m=+6.123157211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret") pod "global-pull-secret-syncer-jw4r5" (UID: "8fd18dc0-806e-46c1-811c-31680ca4fbdb") : object "kube-system"/"original-pull-secret" not registered Jul 28 18:00:59.610107 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.610067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:00:59.610292 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.610238 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 18:00:59.610292 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.610277 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 18:00:59.610292 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.610291 2569 projected.go:196] Error preparing data for projected volume kube-api-access-p2zrd for pod openshift-network-diagnostics/network-check-target-nwkjt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:00:59.610473 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.610356 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd podName:37c30d72-01bf-4c69-a023-cccaa157ef96 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:01.610336688 +0000 UTC m=+6.223621078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2zrd" (UniqueName: "kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd") pod "network-check-target-nwkjt" (UID: "37c30d72-01bf-4c69-a023-cccaa157ef96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:00:59.841795 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.841743 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:55:58 +0000 UTC" deadline="2028-03-29 00:53:20.29286198 +0000 UTC" Jul 28 18:00:59.841795 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.841786 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14622h52m20.451081168s" Jul 28 18:00:59.972289 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.972258 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:00:59.972491 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.972410 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:00:59.972841 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:00:59.972819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:00:59.972973 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:00:59.972951 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:00.969870 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:00.969841 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:00.970352 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:00.969967 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:01.426213 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:01.425663 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:01.426213 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.425812 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:01:01.426213 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.425874 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs podName:c27d9f56-189f-4f1d-bd3b-60fb56978bb3 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:05.425855892 +0000 UTC m=+10.039140284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs") pod "network-metrics-daemon-8z6t7" (UID: "c27d9f56-189f-4f1d-bd3b-60fb56978bb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:01:01.526428 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:01.526391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:01.526660 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.526560 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 18:01:01.526660 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.526630 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret podName:8fd18dc0-806e-46c1-811c-31680ca4fbdb nodeName:}" failed. No retries permitted until 2026-07-28 18:01:05.526610865 +0000 UTC m=+10.139895265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret") pod "global-pull-secret-syncer-jw4r5" (UID: "8fd18dc0-806e-46c1-811c-31680ca4fbdb") : object "kube-system"/"original-pull-secret" not registered Jul 28 18:01:01.627149 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:01.627110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:01.627384 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.627362 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 18:01:01.627467 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.627388 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 18:01:01.627467 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.627402 2569 projected.go:196] Error preparing data for projected volume kube-api-access-p2zrd for pod openshift-network-diagnostics/network-check-target-nwkjt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:01:01.627594 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.627467 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd podName:37c30d72-01bf-4c69-a023-cccaa157ef96 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:05.627448706 +0000 UTC m=+10.240733108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2zrd" (UniqueName: "kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd") pod "network-check-target-nwkjt" (UID: "37c30d72-01bf-4c69-a023-cccaa157ef96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:01:01.972582 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:01.972550 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:01.973025 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.972705 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:01.973103 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:01.973054 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:01.973159 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:01.973142 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:02.970732 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:02.970161 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:02.970732 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:02.970309 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:03.969967 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:03.969932 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:03.970408 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:03.970065 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:03.970408 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:03.969937 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:03.970550 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:03.970528 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:04.970318 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:04.970279 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:04.970805 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:04.970398 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:05.458901 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:05.458864 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:05.459134 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.459110 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:01:05.459269 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.459188 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs podName:c27d9f56-189f-4f1d-bd3b-60fb56978bb3 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:13.459168287 +0000 UTC m=+18.072452691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs") pod "network-metrics-daemon-8z6t7" (UID: "c27d9f56-189f-4f1d-bd3b-60fb56978bb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:01:05.560330 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:05.560291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:05.560517 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.560408 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 18:01:05.560517 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.560463 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret podName:8fd18dc0-806e-46c1-811c-31680ca4fbdb nodeName:}" failed. No retries permitted until 2026-07-28 18:01:13.560448463 +0000 UTC m=+18.173732851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret") pod "global-pull-secret-syncer-jw4r5" (UID: "8fd18dc0-806e-46c1-811c-31680ca4fbdb") : object "kube-system"/"original-pull-secret" not registered Jul 28 18:01:05.661626 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:05.661588 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:05.661801 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.661761 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 18:01:05.661801 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.661789 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 18:01:05.661801 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.661802 2569 projected.go:196] Error preparing data for projected volume kube-api-access-p2zrd for pod openshift-network-diagnostics/network-check-target-nwkjt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:01:05.661961 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.661858 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd podName:37c30d72-01bf-4c69-a023-cccaa157ef96 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:13.661839072 +0000 UTC m=+18.275123463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2zrd" (UniqueName: "kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd") pod "network-check-target-nwkjt" (UID: "37c30d72-01bf-4c69-a023-cccaa157ef96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:01:05.970860 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:05.970829 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:05.971336 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:05.970834 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:05.971336 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.970945 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:05.971336 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:05.971006 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:06.970308 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:06.969766 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:06.970308 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:06.969904 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:07.970457 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:07.970418 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:07.970457 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:07.970454 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:07.970898 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:07.970567 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:07.970898 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:07.970694 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:08.970008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:08.969980 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:08.970151 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:08.970102 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:09.969825 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:09.969788 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:09.970382 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:09.969938 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:09.970382 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:09.969991 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:09.970382 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:09.970096 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:10.970420 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:10.970383 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:10.970870 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:10.970519 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:11.969876 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:11.969835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:11.970043 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:11.969957 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:11.970043 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:11.970019 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:11.970152 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:11.970135 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:12.969585 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:12.969356 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:12.969984 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:12.969669 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:13.516498 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:13.516467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:13.516711 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.516648 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:01:13.516753 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.516734 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs podName:c27d9f56-189f-4f1d-bd3b-60fb56978bb3 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:29.516713013 +0000 UTC m=+34.129997405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs") pod "network-metrics-daemon-8z6t7" (UID: "c27d9f56-189f-4f1d-bd3b-60fb56978bb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:01:13.616953 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:13.616920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:13.617108 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.617031 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 18:01:13.617108 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.617093 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret podName:8fd18dc0-806e-46c1-811c-31680ca4fbdb nodeName:}" failed. No retries permitted until 2026-07-28 18:01:29.617077731 +0000 UTC m=+34.230362137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret") pod "global-pull-secret-syncer-jw4r5" (UID: "8fd18dc0-806e-46c1-811c-31680ca4fbdb") : object "kube-system"/"original-pull-secret" not registered Jul 28 18:01:13.717456 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:13.717413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:13.717645 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.717612 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 18:01:13.717645 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.717640 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 18:01:13.717731 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.717654 2569 projected.go:196] Error preparing data for projected volume kube-api-access-p2zrd for pod openshift-network-diagnostics/network-check-target-nwkjt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:01:13.717731 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.717719 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd podName:37c30d72-01bf-4c69-a023-cccaa157ef96 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:29.717699621 +0000 UTC m=+34.330984015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2zrd" (UniqueName: "kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd") pod "network-check-target-nwkjt" (UID: "37c30d72-01bf-4c69-a023-cccaa157ef96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:01:13.970296 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:13.970258 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:13.970296 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:13.970313 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:13.970766 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.970454 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:13.970766 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:13.970580 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:14.970167 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:14.970127 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:14.970358 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:14.970275 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:15.970501 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:15.970460 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:15.970943 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:15.970600 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:15.970943 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:15.970639 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:15.970943 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:15.970746 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:16.969407 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:16.969372 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:16.969597 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:16.969512 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:17.969870 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:17.969389 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:17.969870 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:17.969520 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:17.970390 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:17.970259 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:17.970390 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:17.970361 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:18.144974 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:18.144936 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"e2dcef17877092f834a4aaeaf2cc3528721bf43610479d49838532737c1ae393"} Jul 28 18:01:18.147530 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:18.147401 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-56lln" event={"ID":"34de2e65-d720-4bdc-b886-978d18c29e1c","Type":"ContainerStarted","Data":"442609daa5cfa30cea7a0c487148bf441c650fa26c32f612e83a8348244989bd"} Jul 28 18:01:18.150171 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:18.150143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" event={"ID":"f02759f6c62b6d84c775f3fa1d163393","Type":"ContainerStarted","Data":"8f9c7c99f58a5e89b9686c676d84103f46d7cdacab7b48cefbfcd34fdf84af24"} Jul 28 18:01:18.151879 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:18.151855 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvms7" event={"ID":"057dfc76-9edb-4849-9f5c-f33869d37b32","Type":"ContainerStarted","Data":"5d35ecf5c013f54487600ca4cf8cd1033d5c9934dac2cef2c4fdd1c42b360af2"} Jul 28 18:01:18.166334 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:18.166291 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-56lln" podStartSLOduration=2.678317339 podStartE2EDuration="22.166277334s" totalImagesPullingTime="19.487959995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:18.165979743 +0000 UTC m=+22.779264154" watchObservedRunningTime="2026-07-28 18:01:18.166277334 +0000 UTC m=+22.779561744" Jul 28 18:01:18.183926 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:18.183887 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pvms7" podStartSLOduration=2.682509821 podStartE2EDuration="22.183873767s" totalImagesPullingTime="19.501363946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:18.183296836 +0000 UTC m=+22.796581245" watchObservedRunningTime="2026-07-28 18:01:18.183873767 +0000 UTC m=+22.797158177" Jul 28 18:01:18.196958 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:18.196922 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" podStartSLOduration=21.196908791 podStartE2EDuration="21.196908791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:18.196627956 +0000 UTC m=+22.809912454" watchObservedRunningTime="2026-07-28 18:01:18.196908791 +0000 UTC m=+22.810193200" Jul 28 18:01:18.969710 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:18.969676 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:18.969906 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:18.969784 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:19.156295 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.156058 2569 generic.go:356] "Generic (PLEG): container finished" podID="c24b599316ee2807955aed8606f79acb" containerID="0fd43ec3588ca62484130e5bc48fceb37f4c89fa8e31760f754d46e13ba8baa2" exitCode=0 Jul 28 18:01:19.156467 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.156153 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"c24b599316ee2807955aed8606f79acb","Type":"ContainerDied","Data":"0fd43ec3588ca62484130e5bc48fceb37f4c89fa8e31760f754d46e13ba8baa2"} Jul 28 18:01:19.158207 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.158168 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d4cgv" event={"ID":"355dd9c8-babd-4952-b249-a5b5c5df07e6","Type":"ContainerStarted","Data":"67420ab7b9549d549e2e55d4597ab38c70f71aec289d7eb02daed262a0041d0e"} Jul 28 18:01:19.163057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.163025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ds5hw" event={"ID":"bb20a3ca-9707-403d-a6d1-6093065b2a46","Type":"ContainerStarted","Data":"4206f49fb04f26d2d87f355fe6b27ac38a3d36e2745616766fe3408ccc9c60be"} Jul 28 18:01:19.164684 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.164658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" event={"ID":"2710a9c1-e24a-41ad-be7a-c87d246c82ad","Type":"ContainerStarted","Data":"9e939e9a5b2ec60061b2a49bcd856b9080af56fcdcbbd2424577846479f95af3"} Jul 28 18:01:19.167636 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.167617 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:01:19.167982 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.167964 2569 generic.go:356] "Generic (PLEG): container finished" podID="8b61e46a-8023-4339-b302-2608710d3964" containerID="e3f69857ccd0afbf424bdcdd63de3386cd30cfeb674d3f2b7bfb1bc8ea710d5c" exitCode=1 Jul 28 18:01:19.168071 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.168042 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"804c8ccfe67ba9272fd826813e5741685d528971e87f81d3ddce9c51b958cb73"} Jul 28 18:01:19.168133 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.168106 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"a249ab0ce5ad50e1daf8dcbe06ab4b7764bf7dd8d62b4e0d57de70638ada57cd"} Jul 28 18:01:19.168184 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.168154 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"c0fb26004d203e046192bead7b470f82194c7f17eb558a4f5fcad099e1949326"} Jul 28 18:01:19.168281 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.168266 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"c47ac66dc5668ffd889d0c7b17bf986a0e889ec30ef8b2c244f448c0a8037364"} Jul 28 18:01:19.168362 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.168320 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerDied","Data":"e3f69857ccd0afbf424bdcdd63de3386cd30cfeb674d3f2b7bfb1bc8ea710d5c"} Jul 28 18:01:19.169737 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.169717 2569 generic.go:356] "Generic (PLEG): container finished" podID="43195a1e-92f0-4c69-bef3-c529d4c75fd8" containerID="29f57d55ef2c0ff8877d6b0983644da62158f65da310bae0b8617222e2133125" exitCode=0 Jul 28 18:01:19.169831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.169790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerDied","Data":"29f57d55ef2c0ff8877d6b0983644da62158f65da310bae0b8617222e2133125"} Jul 28 18:01:19.171452 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.171432 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w829b" event={"ID":"d42d3962-0660-48ee-8248-66d18ead7206","Type":"ContainerStarted","Data":"0d7e4e18df70f68c15bde3d8564c5ef216f02e41150666ea9d0f3eae55fff236"} Jul 28 18:01:19.172989 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.172966 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vn5cx" event={"ID":"62704919-bf62-4605-b9ee-8749f46bb72d","Type":"ContainerStarted","Data":"867c2267397f18605b163726c6a15e1e3f0b2d943b74358ea78ce858b660c275"} Jul 28 18:01:19.187749 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.187710 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d4cgv" podStartSLOduration=3.705083688 podStartE2EDuration="23.187697363s" totalImagesPullingTime="19.482613675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:19.187463252 +0000 UTC m=+23.800747662" watchObservedRunningTime="2026-07-28 18:01:19.187697363 +0000 UTC m=+23.800981806" Jul 28 18:01:19.203887 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.203850 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ds5hw" podStartSLOduration=3.711299281 podStartE2EDuration="23.20383662s" totalImagesPullingTime="19.492537339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:19.203446535 +0000 UTC m=+23.816730948" watchObservedRunningTime="2026-07-28 18:01:19.20383662 +0000 UTC m=+23.817121030" Jul 28 18:01:19.243941 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.243897 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w829b" podStartSLOduration=3.810044697 podStartE2EDuration="23.243882808s" totalImagesPullingTime="19.433838111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:19.243533315 +0000 UTC m=+23.856817726" watchObservedRunningTime="2026-07-28 18:01:19.243882808 +0000 UTC m=+23.857167218" Jul 28 18:01:19.260057 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.260010 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vn5cx" podStartSLOduration=3.793112468 podStartE2EDuration="23.259996204s" totalImagesPullingTime="19.466883736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:19.259369963 +0000 UTC m=+23.872654372" watchObservedRunningTime="2026-07-28 18:01:19.259996204 +0000 UTC m=+23.873280631" Jul 28 18:01:19.723444 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.723342 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 18:01:19.856245 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.856116 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T18:01:19.723354829Z","UUID":"4a18d6ec-59a0-4431-a714-4d76b684d156","Handler":null,"Name":"","Endpoint":""} Jul 28 18:01:19.857790 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.857765 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 18:01:19.857927 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.857810 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 18:01:19.969650 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.969573 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:19.969902 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:19.969573 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:19.969902 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:19.969707 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:19.969902 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:19.969797 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:20.178326 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:20.178269 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"c24b599316ee2807955aed8606f79acb","Type":"ContainerStarted","Data":"8b557abd8454b71df988262289ddae77649064609a21058a15b20b267933c6ca"} Jul 28 18:01:20.180912 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:20.180883 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" event={"ID":"2710a9c1-e24a-41ad-be7a-c87d246c82ad","Type":"ContainerStarted","Data":"18ef2ec757ea7b271522381291025141316038166a1f23937facb437a29514b7"} Jul 28 18:01:20.196838 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:20.196789 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" podStartSLOduration=23.19675537 podStartE2EDuration="23.19675537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:20.196064212 +0000 UTC m=+24.809348624" watchObservedRunningTime="2026-07-28 18:01:20.19675537 +0000 UTC m=+24.810039779" Jul 28 18:01:20.969652 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:20.969618 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:20.969785 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:20.969731 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:21.042811 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:21.042669 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:01:21.043300 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:21.043282 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:01:21.185479 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:21.185443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" event={"ID":"2710a9c1-e24a-41ad-be7a-c87d246c82ad","Type":"ContainerStarted","Data":"33c9d45510816999a903b68f2cfa8c7591f8320577a09b541696a16662cc899a"} Jul 28 18:01:21.188782 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:21.188762 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:01:21.189172 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:21.189134 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"9618775f074494e543b4ef2f9e0bbc9c5093dd0081970a3eb651e6a9eff6e8ce"} Jul 28 18:01:21.206698 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:21.206650 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lq97d" podStartSLOduration=3.944690626 podStartE2EDuration="25.206629714s" totalImagesPullingTime="21.261939088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:21.206366735 +0000 UTC m=+25.819651151" watchObservedRunningTime="2026-07-28 18:01:21.206629714 +0000 UTC m=+25.819914125" Jul 28 18:01:21.969708 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:21.969675 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:21.969890 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:21.969791 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:21.969890 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:21.969834 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:21.969994 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:21.969909 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:22.191186 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:22.191155 2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 28 18:01:22.969829 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:22.969799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:22.970009 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:22.969903 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:23.204676 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:23.204654 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:01:23.205036 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:23.204962 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"84b497a52a36a4d9d33446255eab8301a7cd792678cf4f3a41e8331d6df9aec3"} Jul 28 18:01:23.205286 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:23.205263 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:01:23.205469 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:23.205454 2569 scope.go:117] "RemoveContainer" containerID="e3f69857ccd0afbf424bdcdd63de3386cd30cfeb674d3f2b7bfb1bc8ea710d5c" Jul 28 18:01:23.233187 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:23.233160 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:01:23.969465 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:23.969390 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:23.969642 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:23.969399 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:23.969642 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:23.969510 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:23.969642 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:23.969567 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:24.210513 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.210488 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:01:24.211370 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.210816 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" event={"ID":"8b61e46a-8023-4339-b302-2608710d3964","Type":"ContainerStarted","Data":"fdaf6ed213bf08fb611951987da6e9c860d796c29f5fd1d1da4c7161563707c3"} Jul 28 18:01:24.211370 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.211015 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:01:24.211370 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.211086 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:01:24.212702 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.212679 2569 generic.go:356] "Generic (PLEG): container finished" podID="43195a1e-92f0-4c69-bef3-c529d4c75fd8" containerID="4d39d4a03643459768b4799ddc1c65d1fceabb94820a63bc150d8d2d57074ef6" exitCode=0 Jul 28 18:01:24.212830 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.212718 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerDied","Data":"4d39d4a03643459768b4799ddc1c65d1fceabb94820a63bc150d8d2d57074ef6"} Jul 28 18:01:24.226624 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.226572 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:01:24.245743 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.245704 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" podStartSLOduration=8.4142998 podStartE2EDuration="28.245690461s" totalImagesPullingTime="19.831390661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:24.245479807 +0000 UTC m=+28.858764227" watchObservedRunningTime="2026-07-28 18:01:24.245690461 +0000 UTC m=+28.858974872" Jul 28 18:01:24.969899 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:24.969736 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:24.970139 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:24.970001 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:25.101112 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.101084 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jw4r5"] Jul 28 18:01:25.101309 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.101195 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:25.101309 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:25.101293 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:25.102271 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.102247 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:01:25.102380 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.102361 2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 28 18:01:25.102852 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.102833 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vn5cx" Jul 28 18:01:25.104513 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.104478 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8z6t7"] Jul 28 18:01:25.104729 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.104603 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:25.104729 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:25.104709 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:25.105248 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.105212 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nwkjt"] Jul 28 18:01:25.216875 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.216755 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:25.216875 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:25.216819 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerStarted","Data":"6b7735c7a322a617bf9c45271aeff820e23fbc97005451a5a9f5245ac4814b6f"} Jul 28 18:01:25.216875 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:25.216849 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:26.220980 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:26.220944 2569 generic.go:356] "Generic (PLEG): container finished" podID="43195a1e-92f0-4c69-bef3-c529d4c75fd8" containerID="6b7735c7a322a617bf9c45271aeff820e23fbc97005451a5a9f5245ac4814b6f" exitCode=0 Jul 28 18:01:26.221411 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:26.221031 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerDied","Data":"6b7735c7a322a617bf9c45271aeff820e23fbc97005451a5a9f5245ac4814b6f"} Jul 28 18:01:26.969888 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:26.969854 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:26.970038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:26.969955 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:26.970038 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:26.969966 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:26.970038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:26.969986 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:26.970179 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:26.970055 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:26.970179 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:26.970139 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:28.226999 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:28.226967 2569 generic.go:356] "Generic (PLEG): container finished" podID="43195a1e-92f0-4c69-bef3-c529d4c75fd8" containerID="b119d32ef6c88503a1f4c04b69641ec779103013d159407004011b2a4314bdf8" exitCode=0 Jul 28 18:01:28.227391 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:28.227018 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerDied","Data":"b119d32ef6c88503a1f4c04b69641ec779103013d159407004011b2a4314bdf8"} Jul 28 18:01:28.969943 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:28.969909 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:28.970137 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:28.969998 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:28.970137 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:28.970003 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:28.970137 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:28.970008 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:28.970137 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:28.970103 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:28.970378 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:28.970171 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:29.537933 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:29.537898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:29.538321 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:29.538049 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:01:29.538321 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:29.538120 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs podName:c27d9f56-189f-4f1d-bd3b-60fb56978bb3 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:01.538105841 +0000 UTC m=+66.151390229 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs") pod "network-metrics-daemon-8z6t7" (UID: "c27d9f56-189f-4f1d-bd3b-60fb56978bb3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 18:01:29.638501 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:29.638464 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:29.638650 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:29.638590 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 18:01:29.638650 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:29.638648 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret podName:8fd18dc0-806e-46c1-811c-31680ca4fbdb nodeName:}" failed. No retries permitted until 2026-07-28 18:02:01.638631763 +0000 UTC m=+66.251916171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret") pod "global-pull-secret-syncer-jw4r5" (UID: "8fd18dc0-806e-46c1-811c-31680ca4fbdb") : object "kube-system"/"original-pull-secret" not registered Jul 28 18:01:29.739388 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:29.739316 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:29.739519 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:29.739443 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 18:01:29.739519 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:29.739459 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 18:01:29.739519 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:29.739469 2569 projected.go:196] Error preparing data for projected volume kube-api-access-p2zrd for pod openshift-network-diagnostics/network-check-target-nwkjt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:01:29.739519 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:29.739519 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd podName:37c30d72-01bf-4c69-a023-cccaa157ef96 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:01.739502625 +0000 UTC m=+66.352787036 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2zrd" (UniqueName: "kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd") pod "network-check-target-nwkjt" (UID: "37c30d72-01bf-4c69-a023-cccaa157ef96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 18:01:30.969644 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:30.969436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:30.970050 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:30.969436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:30.970050 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:30.969716 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jw4r5" podUID="8fd18dc0-806e-46c1-811c-31680ca4fbdb" Jul 28 18:01:30.970050 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:30.969815 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8z6t7" podUID="c27d9f56-189f-4f1d-bd3b-60fb56978bb3" Jul 28 18:01:30.970050 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:30.969436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:30.970050 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:30.969894 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nwkjt" podUID="37c30d72-01bf-4c69-a023-cccaa157ef96" Jul 28 18:01:31.691163 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.691136 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeReady" Jul 28 18:01:31.691361 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.691301 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 18:01:31.763119 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.763088 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qdbm7"] Jul 28 18:01:31.764955 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.764935 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9h8xs"] Jul 28 18:01:31.765095 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.765076 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.766958 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.766938 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9h8xs" Jul 28 18:01:31.768610 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.768576 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 18:01:31.769627 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.769608 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9rrw\"" Jul 28 18:01:31.769627 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.769620 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 18:01:31.770009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.769854 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 18:01:31.770009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.769917 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 18:01:31.770009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.769916 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 18:01:31.770197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.770034 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f89dl\"" Jul 28 18:01:31.779673 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.776887 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9h8xs"] Jul 28 18:01:31.782127 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.782106 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdbm7"] Jul 28 18:01:31.855164 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.855129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssnsq\" (UniqueName: \"kubernetes.io/projected/7e4537df-f2f4-438f-9a20-94d0d9728285-kube-api-access-ssnsq\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.855164 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.855167 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579s9\" (UniqueName: \"kubernetes.io/projected/a7f70237-bb77-49a6-8943-8446de31eb9e-kube-api-access-579s9\") pod \"ingress-canary-9h8xs\" (UID: \"a7f70237-bb77-49a6-8943-8446de31eb9e\") " pod="openshift-ingress-canary/ingress-canary-9h8xs" Jul 28 18:01:31.855431 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.855195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4537df-f2f4-438f-9a20-94d0d9728285-tmp-dir\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.855431 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.855248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4537df-f2f4-438f-9a20-94d0d9728285-config-volume\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.855431 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.855273 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.855431 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.855294 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7f70237-bb77-49a6-8943-8446de31eb9e-cert\") pod \"ingress-canary-9h8xs\" (UID: \"a7f70237-bb77-49a6-8943-8446de31eb9e\") " pod="openshift-ingress-canary/ingress-canary-9h8xs" Jul 28 18:01:31.956803 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.956712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssnsq\" (UniqueName: \"kubernetes.io/projected/7e4537df-f2f4-438f-9a20-94d0d9728285-kube-api-access-ssnsq\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.956803 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.956757 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579s9\" (UniqueName: \"kubernetes.io/projected/a7f70237-bb77-49a6-8943-8446de31eb9e-kube-api-access-579s9\") pod \"ingress-canary-9h8xs\" (UID: \"a7f70237-bb77-49a6-8943-8446de31eb9e\") " pod="openshift-ingress-canary/ingress-canary-9h8xs" Jul 28 18:01:31.957040 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.956921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4537df-f2f4-438f-9a20-94d0d9728285-tmp-dir\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.957040 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.956980 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4537df-f2f4-438f-9a20-94d0d9728285-config-volume\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.957040 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.957002 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.957040 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.957031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7f70237-bb77-49a6-8943-8446de31eb9e-cert\") pod \"ingress-canary-9h8xs\" (UID: \"a7f70237-bb77-49a6-8943-8446de31eb9e\") " pod="openshift-ingress-canary/ingress-canary-9h8xs" Jul 28 18:01:31.957270 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:31.957245 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 18:01:31.957341 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:31.957318 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls podName:7e4537df-f2f4-438f-9a20-94d0d9728285 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:32.457296008 +0000 UTC m=+37.070580399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls") pod "dns-default-qdbm7" (UID: "7e4537df-f2f4-438f-9a20-94d0d9728285") : secret "dns-default-metrics-tls" not found Jul 28 18:01:31.957405 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.957350 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4537df-f2f4-438f-9a20-94d0d9728285-tmp-dir\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.957838 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.957815 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4537df-f2f4-438f-9a20-94d0d9728285-config-volume\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.961888 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.961863 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7f70237-bb77-49a6-8943-8446de31eb9e-cert\") pod \"ingress-canary-9h8xs\" (UID: \"a7f70237-bb77-49a6-8943-8446de31eb9e\") " pod="openshift-ingress-canary/ingress-canary-9h8xs" Jul 28 18:01:31.966720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.966697 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssnsq\" (UniqueName: \"kubernetes.io/projected/7e4537df-f2f4-438f-9a20-94d0d9728285-kube-api-access-ssnsq\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:31.966958 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:31.966942 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-579s9\" (UniqueName: \"kubernetes.io/projected/a7f70237-bb77-49a6-8943-8446de31eb9e-kube-api-access-579s9\") pod \"ingress-canary-9h8xs\" (UID: \"a7f70237-bb77-49a6-8943-8446de31eb9e\") " pod="openshift-ingress-canary/ingress-canary-9h8xs" Jul 28 18:01:32.085846 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.085814 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9h8xs" Jul 28 18:01:32.460191 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.460154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:32.460389 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:32.460311 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 18:01:32.460389 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:32.460364 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls podName:7e4537df-f2f4-438f-9a20-94d0d9728285 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:33.460350978 +0000 UTC m=+38.073635366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls") pod "dns-default-qdbm7" (UID: "7e4537df-f2f4-438f-9a20-94d0d9728285") : secret "dns-default-metrics-tls" not found Jul 28 18:01:32.501867 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.501838 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d745bd9bd-lc5q9"] Jul 28 18:01:32.504037 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.504016 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.506553 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.506528 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 18:01:32.506794 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.506775 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 18:01:32.506952 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.506918 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8fv8h\"" Jul 28 18:01:32.507045 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.506975 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 18:01:32.513072 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.513056 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 18:01:32.519729 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.519706 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d745bd9bd-lc5q9"] Jul 28 18:01:32.662354 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.662323 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63041ac7-6ee0-453b-9fcd-baa9c8402682-ca-trust-extracted\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.662354 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.662357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-image-registry-private-configuration\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.662587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.662401 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.662587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.662427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-bound-sa-token\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.662587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.662480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbmt\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-kube-api-access-ddbmt\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.662587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.662547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-trusted-ca\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.662587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.662576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-installation-pull-secrets\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.662776 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.662653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-certificates\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764073 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63041ac7-6ee0-453b-9fcd-baa9c8402682-ca-trust-extracted\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764073 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764047 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-image-registry-private-configuration\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764314 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764093 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764314 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-bound-sa-token\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764314 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbmt\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-kube-api-access-ddbmt\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764314 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-trusted-ca\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764314 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:32.764259 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 18:01:32.764314 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-installation-pull-secrets\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764314 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:32.764279 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d745bd9bd-lc5q9: secret "image-registry-tls" not found Jul 28 18:01:32.764619 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:32.764336 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls podName:63041ac7-6ee0-453b-9fcd-baa9c8402682 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:33.264316147 +0000 UTC m=+37.877600541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls") pod "image-registry-6d745bd9bd-lc5q9" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682") : secret "image-registry-tls" not found Jul 28 18:01:32.764619 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764358 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-certificates\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.764619 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.764417 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63041ac7-6ee0-453b-9fcd-baa9c8402682-ca-trust-extracted\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.765087 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.765062 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-certificates\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.765630 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.765608 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-trusted-ca\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.766859 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.766836 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-installation-pull-secrets\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.767093 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.767075 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-image-registry-private-configuration\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.772727 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.772705 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-bound-sa-token\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.774892 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.774874 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbmt\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-kube-api-access-ddbmt\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:32.970065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.970034 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:01:32.970220 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.970034 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:01:32.970292 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.970034 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:01:32.973447 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.973422 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 18:01:32.973597 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.973489 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9p4qw\"" Jul 28 18:01:32.973728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.973709 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 18:01:32.973828 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.973783 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 18:01:32.974497 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.974472 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x879t\"" Jul 28 18:01:32.974497 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:32.974494 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 18:01:33.269523 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:33.269493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:33.269924 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:33.269673 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 18:01:33.269924 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:33.269693 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d745bd9bd-lc5q9: secret "image-registry-tls" not found Jul 28 18:01:33.269924 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:33.269762 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls podName:63041ac7-6ee0-453b-9fcd-baa9c8402682 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:34.269741024 +0000 UTC m=+38.883025415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls") pod "image-registry-6d745bd9bd-lc5q9" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682") : secret "image-registry-tls" not found Jul 28 18:01:33.394337 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:33.394139 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9h8xs"] Jul 28 18:01:33.398055 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:01:33.398030 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f70237_bb77_49a6_8943_8446de31eb9e.slice/crio-f0d1962febc2776f3d480919319845e3b3a1cb57b90deed357877b8f903ef1c9 WatchSource:0}: Error finding container f0d1962febc2776f3d480919319845e3b3a1cb57b90deed357877b8f903ef1c9: Status 404 returned error can't find the container with id f0d1962febc2776f3d480919319845e3b3a1cb57b90deed357877b8f903ef1c9 Jul 28 18:01:33.471288 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:33.471246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:33.471492 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:33.471410 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 18:01:33.471492 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:33.471476 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls podName:7e4537df-f2f4-438f-9a20-94d0d9728285 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:35.471462527 +0000 UTC m=+40.084746915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls") pod "dns-default-qdbm7" (UID: "7e4537df-f2f4-438f-9a20-94d0d9728285") : secret "dns-default-metrics-tls" not found Jul 28 18:01:34.242165 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:34.242068 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9h8xs" event={"ID":"a7f70237-bb77-49a6-8943-8446de31eb9e","Type":"ContainerStarted","Data":"f0d1962febc2776f3d480919319845e3b3a1cb57b90deed357877b8f903ef1c9"} Jul 28 18:01:34.244394 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:34.244368 2569 generic.go:356] "Generic (PLEG): container finished" podID="43195a1e-92f0-4c69-bef3-c529d4c75fd8" containerID="1e41d5c9eb6c3b118e4ad3a189e6c70ce771bdd1d14423c1db5d38a1dd7b062b" exitCode=0 Jul 28 18:01:34.244519 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:34.244409 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerDied","Data":"1e41d5c9eb6c3b118e4ad3a189e6c70ce771bdd1d14423c1db5d38a1dd7b062b"} Jul 28 18:01:34.276320 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:34.276290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:34.276939 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:34.276505 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 18:01:34.276939 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:34.276524 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d745bd9bd-lc5q9: secret "image-registry-tls" not found Jul 28 18:01:34.277051 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:34.276961 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls podName:63041ac7-6ee0-453b-9fcd-baa9c8402682 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:36.276937499 +0000 UTC m=+40.890221906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls") pod "image-registry-6d745bd9bd-lc5q9" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682") : secret "image-registry-tls" not found Jul 28 18:01:35.160485 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.160450 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq"] Jul 28 18:01:35.164123 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.164101 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.166875 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.166851 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 28 18:01:35.167291 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.167254 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 28 18:01:35.168399 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.168378 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 28 18:01:35.168497 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.168424 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-v72sr\"" Jul 28 18:01:35.168568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.168511 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 28 18:01:35.172197 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.172160 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq"] Jul 28 18:01:35.250572 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.250536 2569 generic.go:356] "Generic (PLEG): container finished" podID="43195a1e-92f0-4c69-bef3-c529d4c75fd8" containerID="17bfeac27aabae44ccf811249e1783be15bd09f70e7eeef4615d56e5db1591ba" exitCode=0 Jul 28 18:01:35.250719 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.250588 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerDied","Data":"17bfeac27aabae44ccf811249e1783be15bd09f70e7eeef4615d56e5db1591ba"} Jul 28 18:01:35.283472 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.283447 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.283834 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.283542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.283834 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.283613 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw5v\" (UniqueName: \"kubernetes.io/projected/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-kube-api-access-lqw5v\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.384474 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.384444 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.384867 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.384834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.384994 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.384921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw5v\" (UniqueName: \"kubernetes.io/projected/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-kube-api-access-lqw5v\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.385359 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.385339 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.388225 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.388204 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.394536 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.394506 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw5v\" (UniqueName: \"kubernetes.io/projected/ec8ee9bc-7ba6-482d-9a6e-4a944173680e-kube-api-access-lqw5v\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq\" (UID: \"ec8ee9bc-7ba6-482d-9a6e-4a944173680e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.475932 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.475859 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" Jul 28 18:01:35.486242 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.486204 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:35.486402 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:35.486381 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 18:01:35.486472 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:35.486456 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls podName:7e4537df-f2f4-438f-9a20-94d0d9728285 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:39.486438533 +0000 UTC m=+44.099722921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls") pod "dns-default-qdbm7" (UID: "7e4537df-f2f4-438f-9a20-94d0d9728285") : secret "dns-default-metrics-tls" not found Jul 28 18:01:35.638503 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:35.638467 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq"] Jul 28 18:01:35.643908 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:01:35.643877 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8ee9bc_7ba6_482d_9a6e_4a944173680e.slice/crio-967f47d2ba576c3c14dca651e3893fe032477ad481a3c69950de0809caf15106 WatchSource:0}: Error finding container 967f47d2ba576c3c14dca651e3893fe032477ad481a3c69950de0809caf15106: Status 404 returned error can't find the container with id 967f47d2ba576c3c14dca651e3893fe032477ad481a3c69950de0809caf15106 Jul 28 18:01:36.167125 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.167089 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp"] Jul 28 18:01:36.194130 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.194100 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp"] Jul 28 18:01:36.194286 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.194250 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp" Jul 28 18:01:36.196999 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.196976 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f5chc\"" Jul 28 18:01:36.258132 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.258098 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84xx7" event={"ID":"43195a1e-92f0-4c69-bef3-c529d4c75fd8","Type":"ContainerStarted","Data":"1e2962c41cfe1003e956b38cee5ec3cabd5ff7dfd661628a5a92d63f2b902539"} Jul 28 18:01:36.259346 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.259318 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" event={"ID":"ec8ee9bc-7ba6-482d-9a6e-4a944173680e","Type":"ContainerStarted","Data":"967f47d2ba576c3c14dca651e3893fe032477ad481a3c69950de0809caf15106"} Jul 28 18:01:36.285207 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.285139 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-84xx7" podStartSLOduration=9.797218975 podStartE2EDuration="40.285118061s" totalImagesPullingTime="30.487899086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:36.284665615 +0000 UTC m=+40.897950027" watchObservedRunningTime="2026-07-28 18:01:36.285118061 +0000 UTC m=+40.898402474" Jul 28 18:01:36.292047 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.292018 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:36.292186 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.292110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9mr\" (UniqueName: \"kubernetes.io/projected/3819dea7-ffd8-4f69-ac80-40478d85d989-kube-api-access-vf9mr\") pod \"network-check-source-7d88cb55d8-7pnpp\" (UID: \"3819dea7-ffd8-4f69-ac80-40478d85d989\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp" Jul 28 18:01:36.292282 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:36.292185 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 18:01:36.292282 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:36.292206 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d745bd9bd-lc5q9: secret "image-registry-tls" not found Jul 28 18:01:36.292383 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:36.292284 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls podName:63041ac7-6ee0-453b-9fcd-baa9c8402682 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:40.292263539 +0000 UTC m=+44.905547926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls") pod "image-registry-6d745bd9bd-lc5q9" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682") : secret "image-registry-tls" not found Jul 28 18:01:36.393106 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.393058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9mr\" (UniqueName: \"kubernetes.io/projected/3819dea7-ffd8-4f69-ac80-40478d85d989-kube-api-access-vf9mr\") pod \"network-check-source-7d88cb55d8-7pnpp\" (UID: \"3819dea7-ffd8-4f69-ac80-40478d85d989\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp" Jul 28 18:01:36.403104 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.403077 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9mr\" (UniqueName: \"kubernetes.io/projected/3819dea7-ffd8-4f69-ac80-40478d85d989-kube-api-access-vf9mr\") pod \"network-check-source-7d88cb55d8-7pnpp\" (UID: \"3819dea7-ffd8-4f69-ac80-40478d85d989\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp" Jul 28 18:01:36.504305 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.504211 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp" Jul 28 18:01:36.637444 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.637412 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lx5c9"] Jul 28 18:01:36.651986 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.651956 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lx5c9"] Jul 28 18:01:36.652153 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.652100 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.654558 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.654531 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4jjmm\"" Jul 28 18:01:36.654782 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.654764 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 18:01:36.654901 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.654814 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 18:01:36.654901 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.654813 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 18:01:36.655041 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.654905 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 18:01:36.696672 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.696643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55c7\" (UniqueName: \"kubernetes.io/projected/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-api-access-v55c7\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.696818 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.696763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27917fe9-74f7-487d-863c-9725cf62b6a4-crio-socket\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.696818 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.696796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.696924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.696834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27917fe9-74f7-487d-863c-9725cf62b6a4-data-volume\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.696924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.696860 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.797831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.797755 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.797831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.797814 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55c7\" (UniqueName: \"kubernetes.io/projected/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-api-access-v55c7\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.798041 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.797872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27917fe9-74f7-487d-863c-9725cf62b6a4-crio-socket\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.798041 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:36.797874 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 18:01:36.798041 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.797888 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.798041 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.797914 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27917fe9-74f7-487d-863c-9725cf62b6a4-data-volume\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.798041 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:36.797952 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:37.297931846 +0000 UTC m=+41.911216239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : configmap "kube-rbac-proxy" not found Jul 28 18:01:36.798041 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:36.798001 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 18:01:36.798314 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:36.798062 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:37.298044082 +0000 UTC m=+41.911328470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : secret "insights-runtime-extractor-tls" not found Jul 28 18:01:36.798314 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.798103 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27917fe9-74f7-487d-863c-9725cf62b6a4-crio-socket\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.798314 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.798186 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27917fe9-74f7-487d-863c-9725cf62b6a4-data-volume\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.811699 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.811678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55c7\" (UniqueName: \"kubernetes.io/projected/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-api-access-v55c7\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:36.995994 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:36.995948 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp"] Jul 28 18:01:37.001213 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:01:37.001185 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3819dea7_ffd8_4f69_ac80_40478d85d989.slice/crio-53e8bf2d39d4be061611bb20e3babd710438d78212a089c8531cf60052919ed9 WatchSource:0}: Error finding container 53e8bf2d39d4be061611bb20e3babd710438d78212a089c8531cf60052919ed9: Status 404 returned error can't find the container with id 53e8bf2d39d4be061611bb20e3babd710438d78212a089c8531cf60052919ed9 Jul 28 18:01:37.264000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.263964 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp" event={"ID":"3819dea7-ffd8-4f69-ac80-40478d85d989","Type":"ContainerStarted","Data":"53e8bf2d39d4be061611bb20e3babd710438d78212a089c8531cf60052919ed9"} Jul 28 18:01:37.265947 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.265628 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9h8xs" event={"ID":"a7f70237-bb77-49a6-8943-8446de31eb9e","Type":"ContainerStarted","Data":"25d865f679a8ab8b2289e534301259177a849be61aef0c2c67a26b553b0f3857"} Jul 28 18:01:37.281213 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.281190 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22"] Jul 28 18:01:37.284381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.284341 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9h8xs" podStartSLOduration=2.678951025 podStartE2EDuration="6.284327148s" totalImagesPullingTime="3.605376123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:37.282990607 +0000 UTC m=+41.896275028" watchObservedRunningTime="2026-07-28 18:01:37.284327148 +0000 UTC m=+41.897611557" Jul 28 18:01:37.302180 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.302150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:37.302595 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.302259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:37.302595 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:37.302371 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 18:01:37.302595 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:37.302433 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:38.302415718 +0000 UTC m=+42.915700128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : configmap "kube-rbac-proxy" not found Jul 28 18:01:37.302977 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:37.302711 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 18:01:37.302977 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:37.302767 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:38.302750517 +0000 UTC m=+42.916034924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : secret "insights-runtime-extractor-tls" not found Jul 28 18:01:37.303322 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.303299 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22"] Jul 28 18:01:37.303436 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.303414 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.306045 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.306025 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 18:01:37.306164 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.306141 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 18:01:37.306482 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.306412 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 18:01:37.306482 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.306460 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 18:01:37.312994 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.308768 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-npbsw\"" Jul 28 18:01:37.402666 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.402630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrpl\" (UniqueName: \"kubernetes.io/projected/14047c71-c3dd-451a-854e-f198c3818fa6-kube-api-access-5wrpl\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.402666 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.402670 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14047c71-c3dd-451a-854e-f198c3818fa6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.402895 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.402692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14047c71-c3dd-451a-854e-f198c3818fa6-config\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.504028 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.503784 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrpl\" (UniqueName: \"kubernetes.io/projected/14047c71-c3dd-451a-854e-f198c3818fa6-kube-api-access-5wrpl\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.504028 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.503843 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14047c71-c3dd-451a-854e-f198c3818fa6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.504028 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.503871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14047c71-c3dd-451a-854e-f198c3818fa6-config\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.504670 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.504647 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14047c71-c3dd-451a-854e-f198c3818fa6-config\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.506712 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.506686 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14047c71-c3dd-451a-854e-f198c3818fa6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.513336 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.513312 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrpl\" (UniqueName: \"kubernetes.io/projected/14047c71-c3dd-451a-854e-f198c3818fa6-kube-api-access-5wrpl\") pod \"service-ca-operator-5d6c75f84f-tgg22\" (UID: \"14047c71-c3dd-451a-854e-f198c3818fa6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.620364 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.620330 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" Jul 28 18:01:37.793673 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:37.793643 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22"] Jul 28 18:01:37.796962 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:01:37.796933 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14047c71_c3dd_451a_854e_f198c3818fa6.slice/crio-57558b742916b10aef47799518a07b1446c8e8dc0ad78fbd11fefefb9c205799 WatchSource:0}: Error finding container 57558b742916b10aef47799518a07b1446c8e8dc0ad78fbd11fefefb9c205799: Status 404 returned error can't find the container with id 57558b742916b10aef47799518a07b1446c8e8dc0ad78fbd11fefefb9c205799 Jul 28 18:01:38.270125 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:38.270086 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" event={"ID":"14047c71-c3dd-451a-854e-f198c3818fa6","Type":"ContainerStarted","Data":"57558b742916b10aef47799518a07b1446c8e8dc0ad78fbd11fefefb9c205799"} Jul 28 18:01:38.310902 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:38.310863 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:38.311414 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:38.310944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:38.311414 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:38.311025 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 18:01:38.311414 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:38.311088 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:40.311070443 +0000 UTC m=+44.924354842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : secret "insights-runtime-extractor-tls" not found Jul 28 18:01:38.311414 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:38.311182 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 18:01:38.311414 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:38.311283 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:40.311262566 +0000 UTC m=+44.924546953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : configmap "kube-rbac-proxy" not found Jul 28 18:01:38.416844 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:38.416731 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w829b_d42d3962-0660-48ee-8248-66d18ead7206/dns-node-resolver/0.log" Jul 28 18:01:39.017843 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:39.017812 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ds5hw_bb20a3ca-9707-403d-a6d1-6093065b2a46/node-ca/0.log" Jul 28 18:01:39.521735 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:39.521699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:39.522188 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:39.521904 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 18:01:39.522188 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:39.521980 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls podName:7e4537df-f2f4-438f-9a20-94d0d9728285 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:47.521957555 +0000 UTC m=+52.135241959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls") pod "dns-default-qdbm7" (UID: "7e4537df-f2f4-438f-9a20-94d0d9728285") : secret "dns-default-metrics-tls" not found Jul 28 18:01:39.817425 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:39.817343 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9h8xs_a7f70237-bb77-49a6-8943-8446de31eb9e/serve-healthcheck-canary/0.log" Jul 28 18:01:40.326759 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:40.326720 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:40.326932 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:40.326814 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:40.326932 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:40.326880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:40.327032 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:40.326979 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 18:01:40.327032 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:40.327000 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d745bd9bd-lc5q9: secret "image-registry-tls" not found Jul 28 18:01:40.327032 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:40.327010 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 18:01:40.327170 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:40.327082 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:44.327060751 +0000 UTC m=+48.940345153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : secret "insights-runtime-extractor-tls" not found Jul 28 18:01:40.327170 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:40.327102 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls podName:63041ac7-6ee0-453b-9fcd-baa9c8402682 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:48.327093415 +0000 UTC m=+52.940377817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls") pod "image-registry-6d745bd9bd-lc5q9" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682") : secret "image-registry-tls" not found Jul 28 18:01:40.327412 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:40.327386 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27917fe9-74f7-487d-863c-9725cf62b6a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:41.280603 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:41.280567 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp" event={"ID":"3819dea7-ffd8-4f69-ac80-40478d85d989","Type":"ContainerStarted","Data":"055f9460944f764dd85dad5d0bbe93078123e21f3487ad181ca5496e72f4f293"} Jul 28 18:01:41.282114 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:41.282085 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" event={"ID":"ec8ee9bc-7ba6-482d-9a6e-4a944173680e","Type":"ContainerStarted","Data":"2d7f0098687e2b38677558285f3822282eaf4fcde0b7a3770b788948bedb912d"} Jul 28 18:01:41.283721 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:41.283699 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" event={"ID":"14047c71-c3dd-451a-854e-f198c3818fa6","Type":"ContainerStarted","Data":"a03bbacbcaca897cbf5408d3cfa5b02e23748e5513c6d70e29213f3b98565df7"} Jul 28 18:01:41.297989 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:41.297936 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7pnpp" podStartSLOduration=1.465493763 podStartE2EDuration="5.29791876s" totalImagesPullingTime="3.832424997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:41.296727432 +0000 UTC m=+45.910011843" watchObservedRunningTime="2026-07-28 18:01:41.29791876 +0000 UTC m=+45.911203177" Jul 28 18:01:41.312943 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:41.312894 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" podStartSLOduration=1.270263709 podStartE2EDuration="4.312875803s" totalImagesPullingTime="3.042612094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:41.312114438 +0000 UTC m=+45.925398851" watchObservedRunningTime="2026-07-28 18:01:41.312875803 +0000 UTC m=+45.926160214" Jul 28 18:01:41.332254 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:41.331988 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" podStartSLOduration=1.154322701 podStartE2EDuration="6.331968993s" totalImagesPullingTime="5.177646292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:41.33195259 +0000 UTC m=+45.945237024" watchObservedRunningTime="2026-07-28 18:01:41.331968993 +0000 UTC m=+45.945253391" Jul 28 18:01:44.363872 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.363829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:44.364347 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:44.363982 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 18:01:44.364347 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:44.364047 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:01:52.364030821 +0000 UTC m=+56.977315209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : secret "insights-runtime-extractor-tls" not found Jul 28 18:01:44.863287 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.863250 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-wtdcm"] Jul 28 18:01:44.892952 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.892919 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-wtdcm"] Jul 28 18:01:44.893107 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.893044 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:44.895916 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.895891 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 28 18:01:44.895916 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.895911 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 28 18:01:44.896067 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.895911 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 28 18:01:44.897055 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.897035 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-r6mlp\"" Jul 28 18:01:44.897176 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.897088 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 28 18:01:44.969736 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.969709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/610d32b8-9b5c-441a-b9dd-d89938ce4111-signing-key\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:44.969891 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.969766 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/610d32b8-9b5c-441a-b9dd-d89938ce4111-signing-cabundle\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:44.969891 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:44.969833 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ww8\" (UniqueName: \"kubernetes.io/projected/610d32b8-9b5c-441a-b9dd-d89938ce4111-kube-api-access-p9ww8\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:45.070740 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:45.070705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/610d32b8-9b5c-441a-b9dd-d89938ce4111-signing-key\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:45.070912 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:45.070763 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/610d32b8-9b5c-441a-b9dd-d89938ce4111-signing-cabundle\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:45.070912 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:45.070806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ww8\" (UniqueName: \"kubernetes.io/projected/610d32b8-9b5c-441a-b9dd-d89938ce4111-kube-api-access-p9ww8\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:45.071531 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:45.071507 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/610d32b8-9b5c-441a-b9dd-d89938ce4111-signing-cabundle\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:45.073061 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:45.073040 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/610d32b8-9b5c-441a-b9dd-d89938ce4111-signing-key\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:45.079588 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:45.079568 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ww8\" (UniqueName: \"kubernetes.io/projected/610d32b8-9b5c-441a-b9dd-d89938ce4111-kube-api-access-p9ww8\") pod \"service-ca-658d7b7c78-wtdcm\" (UID: \"610d32b8-9b5c-441a-b9dd-d89938ce4111\") " pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:45.200863 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:45.200785 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" Jul 28 18:01:45.336786 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:45.336754 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-wtdcm"] Jul 28 18:01:45.340883 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:01:45.340840 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610d32b8_9b5c_441a_b9dd_d89938ce4111.slice/crio-8fec4fd884b31d269ebc0771f1f7fa30d10f9163e93f0bcf2fffc6e09f5ed565 WatchSource:0}: Error finding container 8fec4fd884b31d269ebc0771f1f7fa30d10f9163e93f0bcf2fffc6e09f5ed565: Status 404 returned error can't find the container with id 8fec4fd884b31d269ebc0771f1f7fa30d10f9163e93f0bcf2fffc6e09f5ed565 Jul 28 18:01:46.300282 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:46.300242 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" event={"ID":"610d32b8-9b5c-441a-b9dd-d89938ce4111","Type":"ContainerStarted","Data":"3e256675cda5c33feb9910d212e5c558bdf59512b1c92d48e27857f6fe4a2e42"} Jul 28 18:01:46.300639 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:46.300298 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" event={"ID":"610d32b8-9b5c-441a-b9dd-d89938ce4111","Type":"ContainerStarted","Data":"8fec4fd884b31d269ebc0771f1f7fa30d10f9163e93f0bcf2fffc6e09f5ed565"} Jul 28 18:01:46.316477 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:46.316420 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-wtdcm" podStartSLOduration=2.316400529 podStartE2EDuration="2.316400529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:46.315636426 +0000 UTC m=+50.928920838" watchObservedRunningTime="2026-07-28 18:01:46.316400529 +0000 UTC m=+50.929684939" Jul 28 18:01:47.590340 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:47.590297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:01:47.590830 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:47.590468 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 18:01:47.590830 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:47.590524 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls podName:7e4537df-f2f4-438f-9a20-94d0d9728285 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:03.590507699 +0000 UTC m=+68.203792089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls") pod "dns-default-qdbm7" (UID: "7e4537df-f2f4-438f-9a20-94d0d9728285") : secret "dns-default-metrics-tls" not found Jul 28 18:01:48.397801 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:48.397761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:01:48.397983 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:48.397945 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 18:01:48.397983 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:48.397969 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d745bd9bd-lc5q9: secret "image-registry-tls" not found Jul 28 18:01:48.398071 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:48.398037 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls podName:63041ac7-6ee0-453b-9fcd-baa9c8402682 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:04.398015597 +0000 UTC m=+69.011299989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls") pod "image-registry-6d745bd9bd-lc5q9" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682") : secret "image-registry-tls" not found Jul 28 18:01:52.428120 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:52.428075 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:01:52.428760 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:52.428223 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 18:01:52.428760 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:01:52.428296 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls podName:27917fe9-74f7-487d-863c-9725cf62b6a4 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:08.4282812 +0000 UTC m=+73.041565587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lx5c9" (UID: "27917fe9-74f7-487d-863c-9725cf62b6a4") : secret "insights-runtime-extractor-tls" not found Jul 28 18:01:56.231899 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:01:56.231867 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lxf9" Jul 28 18:02:01.597894 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.597856 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:02:01.600800 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.600781 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 18:02:01.611075 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.611053 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27d9f56-189f-4f1d-bd3b-60fb56978bb3-metrics-certs\") pod \"network-metrics-daemon-8z6t7\" (UID: \"c27d9f56-189f-4f1d-bd3b-60fb56978bb3\") " pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:02:01.698600 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.698545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:02:01.701419 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.701403 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 18:02:01.711700 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.711676 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fd18dc0-806e-46c1-811c-31680ca4fbdb-original-pull-secret\") pod \"global-pull-secret-syncer-jw4r5\" (UID: \"8fd18dc0-806e-46c1-811c-31680ca4fbdb\") " pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:02:01.788580 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.788551 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jw4r5" Jul 28 18:02:01.796479 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.796457 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x879t\"" Jul 28 18:02:01.799351 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.799321 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:02:01.801929 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.801907 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2zrd\" (UniqueName: \"kubernetes.io/projected/37c30d72-01bf-4c69-a023-cccaa157ef96-kube-api-access-p2zrd\") pod \"network-check-target-nwkjt\" (UID: \"37c30d72-01bf-4c69-a023-cccaa157ef96\") " pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:02:01.803447 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.803429 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8z6t7" Jul 28 18:02:01.921213 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.921182 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jw4r5"] Jul 28 18:02:01.924439 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:01.924412 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd18dc0_806e_46c1_811c_31680ca4fbdb.slice/crio-2742638468d74b6b174e3d0380e6985f22c1bb9e3be5c340aaa320a42c84b8c7 WatchSource:0}: Error finding container 2742638468d74b6b174e3d0380e6985f22c1bb9e3be5c340aaa320a42c84b8c7: Status 404 returned error can't find the container with id 2742638468d74b6b174e3d0380e6985f22c1bb9e3be5c340aaa320a42c84b8c7 Jul 28 18:02:01.946635 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:01.946615 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8z6t7"] Jul 28 18:02:01.948385 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:01.948362 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27d9f56_189f_4f1d_bd3b_60fb56978bb3.slice/crio-8e7f0985e5e3c4a57c570d5a517bb2816ef99904b4c2aa0ee22f07b208e99d80 WatchSource:0}: Error finding container 8e7f0985e5e3c4a57c570d5a517bb2816ef99904b4c2aa0ee22f07b208e99d80: Status 404 returned error can't find the container with id 8e7f0985e5e3c4a57c570d5a517bb2816ef99904b4c2aa0ee22f07b208e99d80 Jul 28 18:02:02.084925 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.084899 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9p4qw\"" Jul 28 18:02:02.093150 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.093126 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:02:02.215826 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.215798 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nwkjt"] Jul 28 18:02:02.218398 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:02.218373 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c30d72_01bf_4c69_a023_cccaa157ef96.slice/crio-ec1ff1aa5ac9895a7bb2e68a04e43b575568672166ccd82f30fceebd75ff2361 WatchSource:0}: Error finding container ec1ff1aa5ac9895a7bb2e68a04e43b575568672166ccd82f30fceebd75ff2361: Status 404 returned error can't find the container with id ec1ff1aa5ac9895a7bb2e68a04e43b575568672166ccd82f30fceebd75ff2361 Jul 28 18:02:02.351510 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.351474 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8z6t7" event={"ID":"c27d9f56-189f-4f1d-bd3b-60fb56978bb3","Type":"ContainerStarted","Data":"8e7f0985e5e3c4a57c570d5a517bb2816ef99904b4c2aa0ee22f07b208e99d80"} Jul 28 18:02:02.353331 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.353295 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nwkjt" event={"ID":"37c30d72-01bf-4c69-a023-cccaa157ef96","Type":"ContainerStarted","Data":"460abbe15b40aa2e75e5416859fdec97c227c9b8514b30bedf5bdfa11d2d04f6"} Jul 28 18:02:02.353489 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.353351 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nwkjt" event={"ID":"37c30d72-01bf-4c69-a023-cccaa157ef96","Type":"ContainerStarted","Data":"ec1ff1aa5ac9895a7bb2e68a04e43b575568672166ccd82f30fceebd75ff2361"} Jul 28 18:02:02.353489 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.353376 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:02:02.354492 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.354470 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jw4r5" event={"ID":"8fd18dc0-806e-46c1-811c-31680ca4fbdb","Type":"ContainerStarted","Data":"2742638468d74b6b174e3d0380e6985f22c1bb9e3be5c340aaa320a42c84b8c7"} Jul 28 18:02:02.369530 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:02.369479 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nwkjt" podStartSLOduration=66.369460647 podStartE2EDuration="1m6.369460647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:02.368424866 +0000 UTC m=+66.981709286" watchObservedRunningTime="2026-07-28 18:02:02.369460647 +0000 UTC m=+66.982745058" Jul 28 18:02:03.615357 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:03.615311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:02:03.618206 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:03.618181 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e4537df-f2f4-438f-9a20-94d0d9728285-metrics-tls\") pod \"dns-default-qdbm7\" (UID: \"7e4537df-f2f4-438f-9a20-94d0d9728285\") " pod="openshift-dns/dns-default-qdbm7" Jul 28 18:02:03.880256 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:03.880127 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9rrw\"" Jul 28 18:02:03.888507 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:03.888483 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdbm7" Jul 28 18:02:04.107965 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:04.107837 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdbm7"] Jul 28 18:02:04.423322 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:04.423286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:02:04.425921 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:04.425893 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"image-registry-6d745bd9bd-lc5q9\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:02:04.617622 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:04.617592 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8fv8h\"" Jul 28 18:02:04.624971 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:04.624948 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:02:05.426973 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:05.426936 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4537df_f2f4_438f_9a20_94d0d9728285.slice/crio-6e9284508019de2ecaa3ed716493dfcc99b01bf75a776236e62e64052f6c0807 WatchSource:0}: Error finding container 6e9284508019de2ecaa3ed716493dfcc99b01bf75a776236e62e64052f6c0807: Status 404 returned error can't find the container with id 6e9284508019de2ecaa3ed716493dfcc99b01bf75a776236e62e64052f6c0807 Jul 28 18:02:05.600089 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:05.600066 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d745bd9bd-lc5q9"] Jul 28 18:02:05.603091 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:05.603064 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63041ac7_6ee0_453b_9fcd_baa9c8402682.slice/crio-2513a8766316880c5e66c6de4aa6ceb3a1af8699ab5d2d160d39b5425f9a5bb0 WatchSource:0}: Error finding container 2513a8766316880c5e66c6de4aa6ceb3a1af8699ab5d2d160d39b5425f9a5bb0: Status 404 returned error can't find the container with id 2513a8766316880c5e66c6de4aa6ceb3a1af8699ab5d2d160d39b5425f9a5bb0 Jul 28 18:02:06.370101 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.370052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdbm7" event={"ID":"7e4537df-f2f4-438f-9a20-94d0d9728285","Type":"ContainerStarted","Data":"6e9284508019de2ecaa3ed716493dfcc99b01bf75a776236e62e64052f6c0807"} Jul 28 18:02:06.372162 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.371702 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jw4r5" event={"ID":"8fd18dc0-806e-46c1-811c-31680ca4fbdb","Type":"ContainerStarted","Data":"86a341f945d938ee076fc194954dc87921642005faa82700f7e1ea16438a531b"} Jul 28 18:02:06.373465 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.373436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" event={"ID":"63041ac7-6ee0-453b-9fcd-baa9c8402682","Type":"ContainerStarted","Data":"0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a"} Jul 28 18:02:06.373648 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.373482 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" event={"ID":"63041ac7-6ee0-453b-9fcd-baa9c8402682","Type":"ContainerStarted","Data":"2513a8766316880c5e66c6de4aa6ceb3a1af8699ab5d2d160d39b5425f9a5bb0"} Jul 28 18:02:06.373648 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.373603 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:02:06.375477 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.375451 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8z6t7" event={"ID":"c27d9f56-189f-4f1d-bd3b-60fb56978bb3","Type":"ContainerStarted","Data":"275d1d61213ef65dcf2a0f599b8284b57469ab8caea623610dfdf1d411eae734"} Jul 28 18:02:06.375589 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.375489 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8z6t7" event={"ID":"c27d9f56-189f-4f1d-bd3b-60fb56978bb3","Type":"ContainerStarted","Data":"9d93fd418ba1ee2e1890f67ce683b3385f084bab661d4caf8081a5905ce7574a"} Jul 28 18:02:06.388890 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.388845 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jw4r5" podStartSLOduration=66.488813909 podStartE2EDuration="1m10.388833164s" totalImagesPullingTime="3.900019255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:06.387685694 +0000 UTC m=+71.000970105" watchObservedRunningTime="2026-07-28 18:02:06.388833164 +0000 UTC m=+71.002117574" Jul 28 18:02:06.405011 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.404951 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8z6t7" podStartSLOduration=66.888891652 podStartE2EDuration="1m10.40493369s" totalImagesPullingTime="3.516042038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:06.404930838 +0000 UTC m=+71.018215251" watchObservedRunningTime="2026-07-28 18:02:06.40493369 +0000 UTC m=+71.018218101" Jul 28 18:02:06.425426 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:06.425377 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" podStartSLOduration=34.425359228 podStartE2EDuration="34.425359228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:06.424549488 +0000 UTC m=+71.037833899" watchObservedRunningTime="2026-07-28 18:02:06.425359228 +0000 UTC m=+71.038643635" Jul 28 18:02:08.383240 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.383187 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdbm7" event={"ID":"7e4537df-f2f4-438f-9a20-94d0d9728285","Type":"ContainerStarted","Data":"bcfd4c367d6569ba13b9359369539f1b4c15de441328729ec0b4bb951e2dac6f"} Jul 28 18:02:08.383848 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.383264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdbm7" event={"ID":"7e4537df-f2f4-438f-9a20-94d0d9728285","Type":"ContainerStarted","Data":"d41f39759ba87fc9ad256f10303a3c0041d4bf4f95bf199622a229067aee4cf4"} Jul 28 18:02:08.383848 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.383297 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qdbm7" Jul 28 18:02:08.402890 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.402844 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qdbm7" podStartSLOduration=35.426924306 podStartE2EDuration="37.402829594s" totalImagesPullingTime="1.975905288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:08.40232157 +0000 UTC m=+73.015605985" watchObservedRunningTime="2026-07-28 18:02:08.402829594 +0000 UTC m=+73.016114001" Jul 28 18:02:08.458610 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.458579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:02:08.460872 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.460852 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27917fe9-74f7-487d-863c-9725cf62b6a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx5c9\" (UID: \"27917fe9-74f7-487d-863c-9725cf62b6a4\") " pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:02:08.763346 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.763262 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4jjmm\"" Jul 28 18:02:08.771609 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.771593 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lx5c9" Jul 28 18:02:08.896112 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:08.896081 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lx5c9"] Jul 28 18:02:08.899305 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:08.899276 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27917fe9_74f7_487d_863c_9725cf62b6a4.slice/crio-131dda692ecccf7283f2eb2ea7b88e5d3556847702b5354290161ed12bef334a WatchSource:0}: Error finding container 131dda692ecccf7283f2eb2ea7b88e5d3556847702b5354290161ed12bef334a: Status 404 returned error can't find the container with id 131dda692ecccf7283f2eb2ea7b88e5d3556847702b5354290161ed12bef334a Jul 28 18:02:09.390018 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:09.389907 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx5c9" event={"ID":"27917fe9-74f7-487d-863c-9725cf62b6a4","Type":"ContainerStarted","Data":"131dda692ecccf7283f2eb2ea7b88e5d3556847702b5354290161ed12bef334a"} Jul 28 18:02:10.182279 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.182244 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26"] Jul 28 18:02:10.185621 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.185597 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" Jul 28 18:02:10.189467 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.189440 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-79tp2\"" Jul 28 18:02:10.190306 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.190286 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 28 18:02:10.190450 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.190321 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 28 18:02:10.190450 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.190419 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 28 18:02:10.190557 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.190516 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 28 18:02:10.195922 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.195895 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26"] Jul 28 18:02:10.209742 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.209623 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d745bd9bd-lc5q9"] Jul 28 18:02:10.272563 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.272527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fp5\" (UniqueName: \"kubernetes.io/projected/fd9f7495-33e2-428a-9b83-e2a7d40c7e90-kube-api-access-26fp5\") pod \"managed-serviceaccount-addon-agent-75fb59bb8-j8b26\" (UID: \"fd9f7495-33e2-428a-9b83-e2a7d40c7e90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" Jul 28 18:02:10.272719 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.272581 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd9f7495-33e2-428a-9b83-e2a7d40c7e90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75fb59bb8-j8b26\" (UID: \"fd9f7495-33e2-428a-9b83-e2a7d40c7e90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" Jul 28 18:02:10.275690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.275659 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-62scc"] Jul 28 18:02:10.278749 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.278729 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" Jul 28 18:02:10.281660 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.281639 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cf8zp\"" Jul 28 18:02:10.282605 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.282315 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 28 18:02:10.282605 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.282434 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 28 18:02:10.293208 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.293190 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-62scc"] Jul 28 18:02:10.357030 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.356996 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8675b75794-2vvzv"] Jul 28 18:02:10.360358 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.360337 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.373256 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.373196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fp5\" (UniqueName: \"kubernetes.io/projected/fd9f7495-33e2-428a-9b83-e2a7d40c7e90-kube-api-access-26fp5\") pod \"managed-serviceaccount-addon-agent-75fb59bb8-j8b26\" (UID: \"fd9f7495-33e2-428a-9b83-e2a7d40c7e90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" Jul 28 18:02:10.373518 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.373491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd9f7495-33e2-428a-9b83-e2a7d40c7e90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75fb59bb8-j8b26\" (UID: \"fd9f7495-33e2-428a-9b83-e2a7d40c7e90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" Jul 28 18:02:10.373617 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.373575 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ab1429a6-af73-45b1-acff-7c8c53553881-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-62scc\" (UID: \"ab1429a6-af73-45b1-acff-7c8c53553881\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" Jul 28 18:02:10.373617 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.373601 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab1429a6-af73-45b1-acff-7c8c53553881-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-62scc\" (UID: \"ab1429a6-af73-45b1-acff-7c8c53553881\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" Jul 28 18:02:10.374803 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.374781 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8675b75794-2vvzv"] Jul 28 18:02:10.376698 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.376667 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd9f7495-33e2-428a-9b83-e2a7d40c7e90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75fb59bb8-j8b26\" (UID: \"fd9f7495-33e2-428a-9b83-e2a7d40c7e90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" Jul 28 18:02:10.388915 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.388891 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fp5\" (UniqueName: \"kubernetes.io/projected/fd9f7495-33e2-428a-9b83-e2a7d40c7e90-kube-api-access-26fp5\") pod \"managed-serviceaccount-addon-agent-75fb59bb8-j8b26\" (UID: \"fd9f7495-33e2-428a-9b83-e2a7d40c7e90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" Jul 28 18:02:10.474055 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.473973 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13ad361a-b8bf-4962-9436-bdfa0a101eb8-trusted-ca\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.474055 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474023 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-registry-tls\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.474669 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13ad361a-b8bf-4962-9436-bdfa0a101eb8-ca-trust-extracted\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.474669 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474100 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-bound-sa-token\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.474669 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4499\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-kube-api-access-h4499\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.474669 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474259 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13ad361a-b8bf-4962-9436-bdfa0a101eb8-registry-certificates\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.474669 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ab1429a6-af73-45b1-acff-7c8c53553881-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-62scc\" (UID: \"ab1429a6-af73-45b1-acff-7c8c53553881\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" Jul 28 18:02:10.474669 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab1429a6-af73-45b1-acff-7c8c53553881-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-62scc\" (UID: \"ab1429a6-af73-45b1-acff-7c8c53553881\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" Jul 28 18:02:10.474669 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474333 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13ad361a-b8bf-4962-9436-bdfa0a101eb8-image-registry-private-configuration\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.474669 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.474356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13ad361a-b8bf-4962-9436-bdfa0a101eb8-installation-pull-secrets\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.475043 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.475009 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ab1429a6-af73-45b1-acff-7c8c53553881-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-62scc\" (UID: \"ab1429a6-af73-45b1-acff-7c8c53553881\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" Jul 28 18:02:10.477096 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.477068 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab1429a6-af73-45b1-acff-7c8c53553881-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-62scc\" (UID: \"ab1429a6-af73-45b1-acff-7c8c53553881\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" Jul 28 18:02:10.503132 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.502921 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" Jul 28 18:02:10.575449 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.575417 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13ad361a-b8bf-4962-9436-bdfa0a101eb8-registry-certificates\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.575607 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.575467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13ad361a-b8bf-4962-9436-bdfa0a101eb8-image-registry-private-configuration\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.575607 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.575506 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13ad361a-b8bf-4962-9436-bdfa0a101eb8-installation-pull-secrets\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.575607 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.575535 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13ad361a-b8bf-4962-9436-bdfa0a101eb8-trusted-ca\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.575607 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.575565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-registry-tls\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.575607 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.575595 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13ad361a-b8bf-4962-9436-bdfa0a101eb8-ca-trust-extracted\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.575870 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.575625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-bound-sa-token\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.575870 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.575663 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4499\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-kube-api-access-h4499\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.576572 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.576545 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13ad361a-b8bf-4962-9436-bdfa0a101eb8-registry-certificates\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.576765 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.576709 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13ad361a-b8bf-4962-9436-bdfa0a101eb8-ca-trust-extracted\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.576883 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.576823 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13ad361a-b8bf-4962-9436-bdfa0a101eb8-trusted-ca\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.578747 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.578724 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13ad361a-b8bf-4962-9436-bdfa0a101eb8-installation-pull-secrets\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.579083 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.579060 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-registry-tls\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.579290 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.579269 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13ad361a-b8bf-4962-9436-bdfa0a101eb8-image-registry-private-configuration\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.583910 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.583884 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-bound-sa-token\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.584656 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.584636 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4499\" (UniqueName: \"kubernetes.io/projected/13ad361a-b8bf-4962-9436-bdfa0a101eb8-kube-api-access-h4499\") pod \"image-registry-8675b75794-2vvzv\" (UID: \"13ad361a-b8bf-4962-9436-bdfa0a101eb8\") " pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.590406 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.590381 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" Jul 28 18:02:10.645956 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.645922 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26"] Jul 28 18:02:10.650486 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:10.650452 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd9f7495_33e2_428a_9b83_e2a7d40c7e90.slice/crio-5545d68e7d41f54dbb1f622c8c838c51423c435206ee9680b841d26f82a16443 WatchSource:0}: Error finding container 5545d68e7d41f54dbb1f622c8c838c51423c435206ee9680b841d26f82a16443: Status 404 returned error can't find the container with id 5545d68e7d41f54dbb1f622c8c838c51423c435206ee9680b841d26f82a16443 Jul 28 18:02:10.670686 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.670428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:10.772488 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.772462 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-62scc"] Jul 28 18:02:10.775481 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:10.775439 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1429a6_af73_45b1_acff_7c8c53553881.slice/crio-1de8df5d41f7c3bf84f13c700cf60f4c2c168fdeb89ff47d6a487e3fd85a9790 WatchSource:0}: Error finding container 1de8df5d41f7c3bf84f13c700cf60f4c2c168fdeb89ff47d6a487e3fd85a9790: Status 404 returned error can't find the container with id 1de8df5d41f7c3bf84f13c700cf60f4c2c168fdeb89ff47d6a487e3fd85a9790 Jul 28 18:02:10.822365 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:10.822336 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8675b75794-2vvzv"] Jul 28 18:02:10.824738 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:10.824709 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ad361a_b8bf_4962_9436_bdfa0a101eb8.slice/crio-f1d9a2a022027faa8696a1fb8e16fa7afe8716ee9b26c15a5b921e7a9a00e0ae WatchSource:0}: Error finding container f1d9a2a022027faa8696a1fb8e16fa7afe8716ee9b26c15a5b921e7a9a00e0ae: Status 404 returned error can't find the container with id f1d9a2a022027faa8696a1fb8e16fa7afe8716ee9b26c15a5b921e7a9a00e0ae Jul 28 18:02:11.405171 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:11.405131 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" event={"ID":"ab1429a6-af73-45b1-acff-7c8c53553881","Type":"ContainerStarted","Data":"1de8df5d41f7c3bf84f13c700cf60f4c2c168fdeb89ff47d6a487e3fd85a9790"} Jul 28 18:02:11.407679 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:11.407648 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" event={"ID":"fd9f7495-33e2-428a-9b83-e2a7d40c7e90","Type":"ContainerStarted","Data":"5545d68e7d41f54dbb1f622c8c838c51423c435206ee9680b841d26f82a16443"} Jul 28 18:02:11.411397 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:11.411334 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8675b75794-2vvzv" event={"ID":"13ad361a-b8bf-4962-9436-bdfa0a101eb8","Type":"ContainerStarted","Data":"e56a5829051cc510b28bf06a61dfb994732249e2dd2cc5825f5af5852e88cc82"} Jul 28 18:02:11.411397 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:11.411376 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8675b75794-2vvzv" event={"ID":"13ad361a-b8bf-4962-9436-bdfa0a101eb8","Type":"ContainerStarted","Data":"f1d9a2a022027faa8696a1fb8e16fa7afe8716ee9b26c15a5b921e7a9a00e0ae"} Jul 28 18:02:11.411556 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:11.411537 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:16.435825 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:16.435788 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx5c9" event={"ID":"27917fe9-74f7-487d-863c-9725cf62b6a4","Type":"ContainerStarted","Data":"f67d10781f0e6f4f857685be945ed49e3e3f2c3a5dcde294fdcf6b5da5d9b7b0"} Jul 28 18:02:16.437204 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:16.437166 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" event={"ID":"ab1429a6-af73-45b1-acff-7c8c53553881","Type":"ContainerStarted","Data":"d387093ae3948bb03cf322e4af6db858d807e86f1381c4eed80053f03f95a7e9"} Jul 28 18:02:16.439127 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:16.439105 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" event={"ID":"fd9f7495-33e2-428a-9b83-e2a7d40c7e90","Type":"ContainerStarted","Data":"d2f0bf70cb016302251909f4b79c12f077e28d808d68cad23652476135330ac8"} Jul 28 18:02:16.453446 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:16.453400 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8675b75794-2vvzv" podStartSLOduration=6.45338713 podStartE2EDuration="6.45338713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:11.439862476 +0000 UTC m=+76.053146889" watchObservedRunningTime="2026-07-28 18:02:16.45338713 +0000 UTC m=+81.066671540" Jul 28 18:02:16.454189 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:16.454160 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-62scc" podStartSLOduration=0.918730946 podStartE2EDuration="6.454127687s" totalImagesPullingTime="5.535396741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:16.452774386 +0000 UTC m=+81.066058797" watchObservedRunningTime="2026-07-28 18:02:16.454127687 +0000 UTC m=+81.067412075" Jul 28 18:02:18.393370 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:18.393244 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qdbm7" Jul 28 18:02:18.413358 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:18.413251 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75fb59bb8-j8b26" podStartSLOduration=2.74330413 podStartE2EDuration="8.413212871s" totalImagesPullingTime="5.669908741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:16.468088187 +0000 UTC m=+81.081372598" watchObservedRunningTime="2026-07-28 18:02:18.413212871 +0000 UTC m=+83.026497284" Jul 28 18:02:20.216326 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:20.216251 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:02:20.456340 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:20.456306 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx5c9" event={"ID":"27917fe9-74f7-487d-863c-9725cf62b6a4","Type":"ContainerStarted","Data":"af9681b7b263fce57d2936b07f7b789bd5432f36c8b3c1414503f1033d4c6db2"} Jul 28 18:02:21.015628 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.015595 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct"] Jul 28 18:02:21.019208 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.019149 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.022134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.022112 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 18:02:21.022386 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.022367 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 18:02:21.022497 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.022377 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 18:02:21.023945 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.023357 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 18:02:21.023945 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.023571 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 18:02:21.023945 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.023799 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sh892\"" Jul 28 18:02:21.028795 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.028781 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq"] Jul 28 18:02:21.033783 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.033764 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct"] Jul 28 18:02:21.034670 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.033882 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.037404 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.036797 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 18:02:21.043076 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.042824 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 18:02:21.048780 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.048754 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w65df\"" Jul 28 18:02:21.054124 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.049780 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 18:02:21.054124 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.051716 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zkjtz"] Jul 28 18:02:21.059435 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.059417 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq"] Jul 28 18:02:21.059871 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.059850 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.067406 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.067391 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 18:02:21.067605 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.067445 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 18:02:21.067672 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.067516 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r5f9p\"" Jul 28 18:02:21.067780 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.067552 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 18:02:21.076222 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5lw\" (UniqueName: \"kubernetes.io/projected/ccde900f-54ed-4219-8aca-c86afdc74895-kube-api-access-8r5lw\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.076359 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076283 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.076359 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-sys\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.076464 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076377 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.076464 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-tls\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.076573 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076491 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/27abc741-63d6-450e-a598-84ac1d9a990c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.076573 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-wtmp\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.076573 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076555 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccde900f-54ed-4219-8aca-c86afdc74895-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.076698 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076590 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.076698 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076613 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbp7\" (UniqueName: \"kubernetes.io/projected/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-kube-api-access-qqbp7\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.076698 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076684 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-root\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.076851 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076718 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.076851 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076750 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.076851 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076774 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.076851 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076792 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfxwh\" (UniqueName: \"kubernetes.io/projected/27abc741-63d6-450e-a598-84ac1d9a990c-kube-api-access-zfxwh\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.076851 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-textfile\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.076851 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27abc741-63d6-450e-a598-84ac1d9a990c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.077128 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076883 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-metrics-client-ca\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.077128 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.076907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.178182 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178138 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.178355 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5lw\" (UniqueName: \"kubernetes.io/projected/ccde900f-54ed-4219-8aca-c86afdc74895-kube-api-access-8r5lw\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.178355 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.178355 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-sys\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.178355 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.178355 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-tls\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.178355 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/27abc741-63d6-450e-a598-84ac1d9a990c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.178355 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-wtmp\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccde900f-54ed-4219-8aca-c86afdc74895-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178377 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbp7\" (UniqueName: \"kubernetes.io/projected/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-kube-api-access-qqbp7\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-root\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178437 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178457 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178472 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178488 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfxwh\" (UniqueName: \"kubernetes.io/projected/27abc741-63d6-450e-a598-84ac1d9a990c-kube-api-access-zfxwh\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-textfile\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27abc741-63d6-450e-a598-84ac1d9a990c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.178593 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.178561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-metrics-client-ca\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.179064 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.179047 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-metrics-client-ca\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.179350 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-root\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:02:21.179416 2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:02:21.179484 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-tls podName:ccde900f-54ed-4219-8aca-c86afdc74895 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:21.679466206 +0000 UTC m=+86.292750625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-jlrct" (UID: "ccde900f-54ed-4219-8aca-c86afdc74895") : secret "openshift-state-metrics-tls" not found Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.179637 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:02:21.179722 2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:02:21.179762 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:02:21.179783 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-tls podName:27abc741-63d6-450e-a598-84ac1d9a990c nodeName:}" failed. No retries permitted until 2026-07-28 18:02:21.679764616 +0000 UTC m=+86.293049006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-wskhq" (UID: "27abc741-63d6-450e-a598-84ac1d9a990c") : secret "kube-state-metrics-tls" not found Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:02:21.179802 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-tls podName:c1d0895b-9fc5-44a0-9337-53f0f3aaf32f nodeName:}" failed. No retries permitted until 2026-07-28 18:02:21.679792226 +0000 UTC m=+86.293076620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-tls") pod "node-exporter-zkjtz" (UID: "c1d0895b-9fc5-44a0-9337-53f0f3aaf32f") : secret "node-exporter-tls" not found Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.179801 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-sys\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.179934 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-wtmp\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.180160 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.180190 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-textfile\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.180316 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.180272 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/27abc741-63d6-450e-a598-84ac1d9a990c-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.181024 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.180648 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccde900f-54ed-4219-8aca-c86afdc74895-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.181024 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.180734 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27abc741-63d6-450e-a598-84ac1d9a990c-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.183505 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.183453 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.184144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.184106 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.184650 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.184625 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.192322 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.191028 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbp7\" (UniqueName: \"kubernetes.io/projected/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-kube-api-access-qqbp7\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.192322 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.191567 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5lw\" (UniqueName: \"kubernetes.io/projected/ccde900f-54ed-4219-8aca-c86afdc74895-kube-api-access-8r5lw\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.194029 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.193102 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfxwh\" (UniqueName: \"kubernetes.io/projected/27abc741-63d6-450e-a598-84ac1d9a990c-kube-api-access-zfxwh\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.683492 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.683452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-tls\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.683963 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.683521 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.683963 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.683559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.686367 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.686261 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1d0895b-9fc5-44a0-9337-53f0f3aaf32f-node-exporter-tls\") pod \"node-exporter-zkjtz\" (UID: \"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f\") " pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:21.686571 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.686549 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccde900f-54ed-4219-8aca-c86afdc74895-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jlrct\" (UID: \"ccde900f-54ed-4219-8aca-c86afdc74895\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.686990 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.686967 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/27abc741-63d6-450e-a598-84ac1d9a990c-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wskhq\" (UID: \"27abc741-63d6-450e-a598-84ac1d9a990c\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.936380 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.936300 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" Jul 28 18:02:21.958025 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.958002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" Jul 28 18:02:21.971428 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:21.971408 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zkjtz" Jul 28 18:02:23.415477 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:23.415445 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d0895b_9fc5_44a0_9337_53f0f3aaf32f.slice/crio-e78d4e500f879b828422f4bf99b8924e9ffdffbd8f28c1cc6fda067130f39f98 WatchSource:0}: Error finding container e78d4e500f879b828422f4bf99b8924e9ffdffbd8f28c1cc6fda067130f39f98: Status 404 returned error can't find the container with id e78d4e500f879b828422f4bf99b8924e9ffdffbd8f28c1cc6fda067130f39f98 Jul 28 18:02:23.471317 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:23.470991 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkjtz" event={"ID":"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f","Type":"ContainerStarted","Data":"e78d4e500f879b828422f4bf99b8924e9ffdffbd8f28c1cc6fda067130f39f98"} Jul 28 18:02:23.551995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:23.551960 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct"] Jul 28 18:02:23.556001 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:23.555976 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccde900f_54ed_4219_8aca_c86afdc74895.slice/crio-d003c5b073b57486bd82afa791a20080fe7539d530099bf6f9226ffd384bb2ef WatchSource:0}: Error finding container d003c5b073b57486bd82afa791a20080fe7539d530099bf6f9226ffd384bb2ef: Status 404 returned error can't find the container with id d003c5b073b57486bd82afa791a20080fe7539d530099bf6f9226ffd384bb2ef Jul 28 18:02:23.566102 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:23.566082 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq"] Jul 28 18:02:23.568129 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:23.568106 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27abc741_63d6_450e_a598_84ac1d9a990c.slice/crio-29e055a6d2aaef1c39a57b4d3c11fcdb98a61f47b80328f56d847f54ee94517f WatchSource:0}: Error finding container 29e055a6d2aaef1c39a57b4d3c11fcdb98a61f47b80328f56d847f54ee94517f: Status 404 returned error can't find the container with id 29e055a6d2aaef1c39a57b4d3c11fcdb98a61f47b80328f56d847f54ee94517f Jul 28 18:02:24.042886 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.039835 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-f694f7dcd-gqdrn"] Jul 28 18:02:24.045353 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.045320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.048105 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.048076 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 18:02:24.048263 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.048221 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dso8a4k4pu7ho\"" Jul 28 18:02:24.048345 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.048292 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 18:02:24.048403 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.048348 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 18:02:24.048540 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.048521 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 18:02:24.048540 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.048528 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rs5bh\"" Jul 28 18:02:24.048540 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.048543 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 18:02:24.054712 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.054688 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f694f7dcd-gqdrn"] Jul 28 18:02:24.102027 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.101998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-grpc-tls\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.102205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.102036 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.102205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.102061 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltw4\" (UniqueName: \"kubernetes.io/projected/f425edc9-5a21-4128-8067-0563c2c83e1e-kube-api-access-dltw4\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.102205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.102155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f425edc9-5a21-4128-8067-0563c2c83e1e-metrics-client-ca\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.102410 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.102209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-tls\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.102410 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.102281 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.102410 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.102320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.102410 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.102343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.203327 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.203290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.203499 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.203341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.203499 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.203372 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.203499 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.203445 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-grpc-tls\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.203499 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.203475 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.203697 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.203507 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltw4\" (UniqueName: \"kubernetes.io/projected/f425edc9-5a21-4128-8067-0563c2c83e1e-kube-api-access-dltw4\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.203697 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.203539 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f425edc9-5a21-4128-8067-0563c2c83e1e-metrics-client-ca\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.203697 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.203569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-tls\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.205917 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.205821 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f425edc9-5a21-4128-8067-0563c2c83e1e-metrics-client-ca\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.206748 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.206701 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-tls\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.207466 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.207442 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.208459 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.208419 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.208557 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.208534 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.208858 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.208817 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-grpc-tls\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.209295 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.209272 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f425edc9-5a21-4128-8067-0563c2c83e1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.214552 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.214504 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltw4\" (UniqueName: \"kubernetes.io/projected/f425edc9-5a21-4128-8067-0563c2c83e1e-kube-api-access-dltw4\") pod \"thanos-querier-f694f7dcd-gqdrn\" (UID: \"f425edc9-5a21-4128-8067-0563c2c83e1e\") " pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.358640 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.358372 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:24.476698 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.476657 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" event={"ID":"27abc741-63d6-450e-a598-84ac1d9a990c","Type":"ContainerStarted","Data":"29e055a6d2aaef1c39a57b4d3c11fcdb98a61f47b80328f56d847f54ee94517f"} Jul 28 18:02:24.480307 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.479782 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx5c9" event={"ID":"27917fe9-74f7-487d-863c-9725cf62b6a4","Type":"ContainerStarted","Data":"d942da8ce7210e5404db8318afb692ff79b15839719f6d10708575eff3131655"} Jul 28 18:02:24.482058 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.482032 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" event={"ID":"ccde900f-54ed-4219-8aca-c86afdc74895","Type":"ContainerStarted","Data":"ea098fea992a3fc554f1886b153a427b2fbf3cb5ac570170124a5bdd5d277eaf"} Jul 28 18:02:24.482183 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.482078 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" event={"ID":"ccde900f-54ed-4219-8aca-c86afdc74895","Type":"ContainerStarted","Data":"21b609c4ddec231ef1781ac89bdaa61703231f5b55e1b3e3e23a92d8e5be5d77"} Jul 28 18:02:24.482183 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.482099 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" event={"ID":"ccde900f-54ed-4219-8aca-c86afdc74895","Type":"ContainerStarted","Data":"d003c5b073b57486bd82afa791a20080fe7539d530099bf6f9226ffd384bb2ef"} Jul 28 18:02:24.498241 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.498175 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lx5c9" podStartSLOduration=34.053598887 podStartE2EDuration="48.498146942s" totalImagesPullingTime="14.444548055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:24.496770007 +0000 UTC m=+89.110054441" watchObservedRunningTime="2026-07-28 18:02:24.498146942 +0000 UTC m=+89.111431353" Jul 28 18:02:24.612326 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:24.612241 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f694f7dcd-gqdrn"] Jul 28 18:02:24.724256 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:24.724210 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf425edc9_5a21_4128_8067_0563c2c83e1e.slice/crio-03605f06d51b472c25f8dd3a240a87f2e1946b18665fe07890bd8b4366b5a428 WatchSource:0}: Error finding container 03605f06d51b472c25f8dd3a240a87f2e1946b18665fe07890bd8b4366b5a428: Status 404 returned error can't find the container with id 03605f06d51b472c25f8dd3a240a87f2e1946b18665fe07890bd8b4366b5a428 Jul 28 18:02:25.487739 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.487703 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" event={"ID":"27abc741-63d6-450e-a598-84ac1d9a990c","Type":"ContainerStarted","Data":"8414a32b2aaf67f1fd191e45b8dc17097ca8bd707e4a47382093070ceff9ed39"} Jul 28 18:02:25.488333 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.487770 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" event={"ID":"27abc741-63d6-450e-a598-84ac1d9a990c","Type":"ContainerStarted","Data":"ab2586c69f992d1e88d909ccd1464955c74859f97aeb91d8c291749b994eb5ce"} Jul 28 18:02:25.488333 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.487804 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" event={"ID":"27abc741-63d6-450e-a598-84ac1d9a990c","Type":"ContainerStarted","Data":"55e331b9ffbcd2cc77fc1c84971eb1783737efa461ab739ddc76ff9bbe9456b2"} Jul 28 18:02:25.489129 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.489098 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" event={"ID":"f425edc9-5a21-4128-8067-0563c2c83e1e","Type":"ContainerStarted","Data":"03605f06d51b472c25f8dd3a240a87f2e1946b18665fe07890bd8b4366b5a428"} Jul 28 18:02:25.490610 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.490587 2569 generic.go:356] "Generic (PLEG): container finished" podID="c1d0895b-9fc5-44a0-9337-53f0f3aaf32f" containerID="e61f7895a91e5bfd81f4abb21900732543d69030373a48b6a94c319a69445dee" exitCode=0 Jul 28 18:02:25.490725 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.490675 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkjtz" event={"ID":"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f","Type":"ContainerDied","Data":"e61f7895a91e5bfd81f4abb21900732543d69030373a48b6a94c319a69445dee"} Jul 28 18:02:25.492695 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.492674 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" event={"ID":"ccde900f-54ed-4219-8aca-c86afdc74895","Type":"ContainerStarted","Data":"a6d44759a37a63eed0b568cecd02f97c605e314280bb95c65d78169b349d6e1d"} Jul 28 18:02:25.528297 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.528251 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wskhq" podStartSLOduration=3.013353975 podStartE2EDuration="4.528216652s" totalImagesPullingTime="1.514862677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:25.525891376 +0000 UTC m=+90.139175786" watchObservedRunningTime="2026-07-28 18:02:25.528216652 +0000 UTC m=+90.141501062" Jul 28 18:02:25.547205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:25.547149 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jlrct" podStartSLOduration=4.131006683 podStartE2EDuration="5.547127652s" totalImagesPullingTime="1.416120969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:25.545983696 +0000 UTC m=+90.159268106" watchObservedRunningTime="2026-07-28 18:02:25.547127652 +0000 UTC m=+90.160412068" Jul 28 18:02:26.500169 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:26.500124 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkjtz" event={"ID":"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f","Type":"ContainerStarted","Data":"80180d211b32686f94c5105a1f82fa6fac1c0c2736b26003c451ca383df5cd41"} Jul 28 18:02:26.500651 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:26.500204 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkjtz" event={"ID":"c1d0895b-9fc5-44a0-9337-53f0f3aaf32f","Type":"ContainerStarted","Data":"7b69619e750fac25076658196171c14b6cbba9a21540116b69f70de2558dd14a"} Jul 28 18:02:26.526631 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:26.526510 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zkjtz" podStartSLOduration=3.786056295 podStartE2EDuration="5.526491676s" totalImagesPullingTime="1.660623812s" totalInitContainerRuntime="79.811569ms" isStatefulPod=true podCreationTimestamp="2026-07-28 18:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:26.525094957 +0000 UTC m=+91.138379369" watchObservedRunningTime="2026-07-28 18:02:26.526491676 +0000 UTC m=+91.139776087" Jul 28 18:02:27.505568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.505534 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" event={"ID":"f425edc9-5a21-4128-8067-0563c2c83e1e","Type":"ContainerStarted","Data":"9f66a45929ef262999db094e4a8669afd447ecaeca8b73336457eb3c66bf1124"} Jul 28 18:02:27.505885 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.505600 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" event={"ID":"f425edc9-5a21-4128-8067-0563c2c83e1e","Type":"ContainerStarted","Data":"2985e6d61d91123165777769d6196b91129048260e3c43982b369672cd3c25d7"} Jul 28 18:02:27.505885 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.505629 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" event={"ID":"f425edc9-5a21-4128-8067-0563c2c83e1e","Type":"ContainerStarted","Data":"9b212f4680cf377af6d96a8aa3ac5c41ddfa9cf362ca2c3e86dad534cd1b9241"} Jul 28 18:02:27.680391 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.680358 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-c9579"] Jul 28 18:02:27.686188 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.686151 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-c9579" Jul 28 18:02:27.688970 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.688948 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 18:02:27.689108 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.689017 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 18:02:27.689108 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.689085 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-72nnq\"" Jul 28 18:02:27.694282 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.694259 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-c9579"] Jul 28 18:02:27.735312 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.735276 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mln\" (UniqueName: \"kubernetes.io/projected/aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e-kube-api-access-z9mln\") pod \"downloads-7c97dc4555-c9579\" (UID: \"aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e\") " pod="openshift-console/downloads-7c97dc4555-c9579" Jul 28 18:02:27.835667 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.835629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mln\" (UniqueName: \"kubernetes.io/projected/aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e-kube-api-access-z9mln\") pod \"downloads-7c97dc4555-c9579\" (UID: \"aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e\") " pod="openshift-console/downloads-7c97dc4555-c9579" Jul 28 18:02:27.846351 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.846329 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mln\" (UniqueName: \"kubernetes.io/projected/aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e-kube-api-access-z9mln\") pod \"downloads-7c97dc4555-c9579\" (UID: \"aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e\") " pod="openshift-console/downloads-7c97dc4555-c9579" Jul 28 18:02:27.996541 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:27.996467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-c9579" Jul 28 18:02:28.221656 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:28.221624 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-c9579"] Jul 28 18:02:28.223685 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:02:28.223654 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa052ba4_aa2e_42ff_9ec8_ce2472d04d2e.slice/crio-c036cdf897fdd88d471998e2f12075160f82191f6fd6bf764fd316253b6472f8 WatchSource:0}: Error finding container c036cdf897fdd88d471998e2f12075160f82191f6fd6bf764fd316253b6472f8: Status 404 returned error can't find the container with id c036cdf897fdd88d471998e2f12075160f82191f6fd6bf764fd316253b6472f8 Jul 28 18:02:28.512298 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:28.512181 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" event={"ID":"f425edc9-5a21-4128-8067-0563c2c83e1e","Type":"ContainerStarted","Data":"ab0a6d755c483d36aedc8e2a7897eb618f92366bcfab0880ddd02c5e32eace25"} Jul 28 18:02:28.512298 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:28.512276 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" event={"ID":"f425edc9-5a21-4128-8067-0563c2c83e1e","Type":"ContainerStarted","Data":"3707c5a188a3bfdf3c2a8e77419c6ea581a44da9e193180255b27ef29471bf64"} Jul 28 18:02:28.512784 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:28.512305 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" event={"ID":"f425edc9-5a21-4128-8067-0563c2c83e1e","Type":"ContainerStarted","Data":"a75bc3d5a3702538878826e7413c20e477e683f026143b208ffae9001c91a05b"} Jul 28 18:02:28.513428 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:28.513407 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-c9579" event={"ID":"aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e","Type":"ContainerStarted","Data":"c036cdf897fdd88d471998e2f12075160f82191f6fd6bf764fd316253b6472f8"} Jul 28 18:02:28.536732 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:28.536677 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" podStartSLOduration=1.302899351 podStartE2EDuration="4.536657593s" totalImagesPullingTime="3.233758242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:28.534659308 +0000 UTC m=+93.147943717" watchObservedRunningTime="2026-07-28 18:02:28.536657593 +0000 UTC m=+93.149942004" Jul 28 18:02:29.518887 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:29.518861 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:32.420579 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:32.420550 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8675b75794-2vvzv" Jul 28 18:02:33.360831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:33.360798 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nwkjt" Jul 28 18:02:35.231176 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.231128 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" podUID="63041ac7-6ee0-453b-9fcd-baa9c8402682" containerName="registry" containerID="cri-o://0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a" gracePeriod=30 Jul 28 18:02:35.485822 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.485767 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:02:35.529739 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.529709 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f694f7dcd-gqdrn" Jul 28 18:02:35.546505 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.546478 2569 generic.go:356] "Generic (PLEG): container finished" podID="63041ac7-6ee0-453b-9fcd-baa9c8402682" containerID="0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a" exitCode=0 Jul 28 18:02:35.546665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.546531 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" Jul 28 18:02:35.546665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.546569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" event={"ID":"63041ac7-6ee0-453b-9fcd-baa9c8402682","Type":"ContainerDied","Data":"0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a"} Jul 28 18:02:35.546665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.546630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d745bd9bd-lc5q9" event={"ID":"63041ac7-6ee0-453b-9fcd-baa9c8402682","Type":"ContainerDied","Data":"2513a8766316880c5e66c6de4aa6ceb3a1af8699ab5d2d160d39b5425f9a5bb0"} Jul 28 18:02:35.546665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.546658 2569 scope.go:117] "RemoveContainer" containerID="0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a" Jul 28 18:02:35.556259 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.556222 2569 scope.go:117] "RemoveContainer" containerID="0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a" Jul 28 18:02:35.556558 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:02:35.556527 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a\": container with ID starting with 0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a not found: ID does not exist" containerID="0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a" Jul 28 18:02:35.556676 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.556568 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a"} err="failed to get container status \"0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a\": rpc error: code = NotFound desc = could not find container \"0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a\": container with ID starting with 0a74379fb8af66338ed5e5ed0a638bf74b677076b79eee4676a410fcfa782f8a not found: ID does not exist" Jul 28 18:02:35.605077 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605049 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63041ac7-6ee0-453b-9fcd-baa9c8402682-ca-trust-extracted\") pod \"63041ac7-6ee0-453b-9fcd-baa9c8402682\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " Jul 28 18:02:35.605264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605095 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-installation-pull-secrets\") pod \"63041ac7-6ee0-453b-9fcd-baa9c8402682\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " Jul 28 18:02:35.605264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605133 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") pod \"63041ac7-6ee0-453b-9fcd-baa9c8402682\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " Jul 28 18:02:35.605264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605155 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddbmt\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-kube-api-access-ddbmt\") pod \"63041ac7-6ee0-453b-9fcd-baa9c8402682\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " Jul 28 18:02:35.605264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605178 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-trusted-ca\") pod \"63041ac7-6ee0-453b-9fcd-baa9c8402682\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " Jul 28 18:02:35.605264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605211 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-image-registry-private-configuration\") pod \"63041ac7-6ee0-453b-9fcd-baa9c8402682\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " Jul 28 18:02:35.605758 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605729 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-certificates\") pod \"63041ac7-6ee0-453b-9fcd-baa9c8402682\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " Jul 28 18:02:35.605836 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605760 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "63041ac7-6ee0-453b-9fcd-baa9c8402682" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:02:35.605836 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.605805 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-bound-sa-token\") pod \"63041ac7-6ee0-453b-9fcd-baa9c8402682\" (UID: \"63041ac7-6ee0-453b-9fcd-baa9c8402682\") " Jul 28 18:02:35.606211 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.606192 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-trusted-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:02:35.607138 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.606822 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "63041ac7-6ee0-453b-9fcd-baa9c8402682" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:02:35.609140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.609113 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "63041ac7-6ee0-453b-9fcd-baa9c8402682" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:02:35.609321 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.609299 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "63041ac7-6ee0-453b-9fcd-baa9c8402682" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:02:35.609408 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.609342 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-kube-api-access-ddbmt" (OuterVolumeSpecName: "kube-api-access-ddbmt") pod "63041ac7-6ee0-453b-9fcd-baa9c8402682" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682"). InnerVolumeSpecName "kube-api-access-ddbmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:02:35.609463 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.609449 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "63041ac7-6ee0-453b-9fcd-baa9c8402682" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:02:35.609517 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.609507 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "63041ac7-6ee0-453b-9fcd-baa9c8402682" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:02:35.616888 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.616865 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63041ac7-6ee0-453b-9fcd-baa9c8402682-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "63041ac7-6ee0-453b-9fcd-baa9c8402682" (UID: "63041ac7-6ee0-453b-9fcd-baa9c8402682"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:02:35.706891 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.706851 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-certificates\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:02:35.706891 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.706891 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-bound-sa-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:02:35.707094 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.706904 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63041ac7-6ee0-453b-9fcd-baa9c8402682-ca-trust-extracted\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:02:35.707094 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.706916 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-installation-pull-secrets\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:02:35.707094 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.706930 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-registry-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:02:35.707094 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.706942 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddbmt\" (UniqueName: \"kubernetes.io/projected/63041ac7-6ee0-453b-9fcd-baa9c8402682-kube-api-access-ddbmt\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:02:35.707094 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.706953 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63041ac7-6ee0-453b-9fcd-baa9c8402682-image-registry-private-configuration\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:02:35.874935 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.874884 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d745bd9bd-lc5q9"] Jul 28 18:02:35.877603 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.877577 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d745bd9bd-lc5q9"] Jul 28 18:02:35.975821 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:35.975783 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63041ac7-6ee0-453b-9fcd-baa9c8402682" path="/var/lib/kubelet/pods/63041ac7-6ee0-453b-9fcd-baa9c8402682/volumes" Jul 28 18:02:46.599955 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:46.599908 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-c9579" event={"ID":"aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e","Type":"ContainerStarted","Data":"ba7333c7818248774723e2736970b9960e9095e900a2d339be8afd78572fd2e1"} Jul 28 18:02:46.600504 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:46.600286 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-c9579" Jul 28 18:02:46.614702 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:46.614676 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-c9579" Jul 28 18:02:46.622508 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:46.622453 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-c9579" podStartSLOduration=2.257214668 podStartE2EDuration="19.622436557s" totalImagesPullingTime="17.365221889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:46.620142222 +0000 UTC m=+111.233426644" watchObservedRunningTime="2026-07-28 18:02:46.622436557 +0000 UTC m=+111.235720966" Jul 28 18:02:56.641685 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:56.641650 2569 generic.go:356] "Generic (PLEG): container finished" podID="ec8ee9bc-7ba6-482d-9a6e-4a944173680e" containerID="2d7f0098687e2b38677558285f3822282eaf4fcde0b7a3770b788948bedb912d" exitCode=0 Jul 28 18:02:56.642148 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:56.641727 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" event={"ID":"ec8ee9bc-7ba6-482d-9a6e-4a944173680e","Type":"ContainerDied","Data":"2d7f0098687e2b38677558285f3822282eaf4fcde0b7a3770b788948bedb912d"} Jul 28 18:02:56.642148 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:56.642106 2569 scope.go:117] "RemoveContainer" containerID="2d7f0098687e2b38677558285f3822282eaf4fcde0b7a3770b788948bedb912d" Jul 28 18:02:56.643292 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:56.643253 2569 generic.go:356] "Generic (PLEG): container finished" podID="14047c71-c3dd-451a-854e-f198c3818fa6" containerID="a03bbacbcaca897cbf5408d3cfa5b02e23748e5513c6d70e29213f3b98565df7" exitCode=0 Jul 28 18:02:56.643387 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:56.643318 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" event={"ID":"14047c71-c3dd-451a-854e-f198c3818fa6","Type":"ContainerDied","Data":"a03bbacbcaca897cbf5408d3cfa5b02e23748e5513c6d70e29213f3b98565df7"} Jul 28 18:02:56.643607 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:56.643595 2569 scope.go:117] "RemoveContainer" containerID="a03bbacbcaca897cbf5408d3cfa5b02e23748e5513c6d70e29213f3b98565df7" Jul 28 18:02:57.649124 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:57.649083 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq" event={"ID":"ec8ee9bc-7ba6-482d-9a6e-4a944173680e","Type":"ContainerStarted","Data":"9a5324b3078947637cc5fa42fc1867966bce6961fbbb06a8bdfb0298f60d1216"} Jul 28 18:02:57.650751 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:02:57.650728 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tgg22" event={"ID":"14047c71-c3dd-451a-854e-f198c3818fa6","Type":"ContainerStarted","Data":"e603b8f9e6c09b2eca5438700c46a13970010468e7593019cb6fac8ae42bb3ea"} Jul 28 18:03:45.374164 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.374124 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6"] Jul 28 18:03:45.374628 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.374465 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63041ac7-6ee0-453b-9fcd-baa9c8402682" containerName="registry" Jul 28 18:03:45.374628 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.374477 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="63041ac7-6ee0-453b-9fcd-baa9c8402682" containerName="registry" Jul 28 18:03:45.374628 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.374527 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="63041ac7-6ee0-453b-9fcd-baa9c8402682" containerName="registry" Jul 28 18:03:45.377427 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.377406 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.380378 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.380357 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 18:03:45.380904 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.380886 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 18:03:45.382144 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.382129 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 18:03:45.382505 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.382485 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kphfc\"" Jul 28 18:03:45.382505 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.382498 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 18:03:45.382667 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.382498 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 18:03:45.386463 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.386447 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 18:03:45.395455 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.395433 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6"] Jul 28 18:03:45.507044 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.507009 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjlq\" (UniqueName: \"kubernetes.io/projected/f89be472-d3aa-44a5-8d22-068bdea37f86-kube-api-access-ctjlq\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.507213 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.507060 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-telemeter-client-tls\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.507213 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.507117 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.507213 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.507171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-metrics-client-ca\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.507213 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.507198 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.507375 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.507273 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-serving-certs-ca-bundle\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.507375 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.507292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-secret-telemeter-client\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.507375 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.507316 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-federate-client-tls\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.608280 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.608223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-metrics-client-ca\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.608430 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.608284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.608430 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.608315 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-serving-certs-ca-bundle\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.608430 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.608334 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-secret-telemeter-client\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.608430 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.608359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-federate-client-tls\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.608430 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.608379 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjlq\" (UniqueName: \"kubernetes.io/projected/f89be472-d3aa-44a5-8d22-068bdea37f86-kube-api-access-ctjlq\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.608634 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.608435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-telemeter-client-tls\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.608634 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.608461 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.609130 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.609103 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-metrics-client-ca\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.609267 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.609125 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-serving-certs-ca-bundle\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.609612 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.609589 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89be472-d3aa-44a5-8d22-068bdea37f86-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.610873 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.610848 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-secret-telemeter-client\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.610968 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.610917 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.611070 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.611052 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-telemeter-client-tls\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.611359 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.611338 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f89be472-d3aa-44a5-8d22-068bdea37f86-federate-client-tls\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.616737 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.616717 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjlq\" (UniqueName: \"kubernetes.io/projected/f89be472-d3aa-44a5-8d22-068bdea37f86-kube-api-access-ctjlq\") pod \"telemeter-client-79bd4b8bff-7vqc6\" (UID: \"f89be472-d3aa-44a5-8d22-068bdea37f86\") " pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.687185 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.687105 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" Jul 28 18:03:45.827984 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.827938 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6"] Jul 28 18:03:45.838291 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:45.838261 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" event={"ID":"f89be472-d3aa-44a5-8d22-068bdea37f86","Type":"ContainerStarted","Data":"c0feb117ee20fb9cfb3fdd8481c3fbf724cfa54798ce565c23db59505dcb20fa"} Jul 28 18:03:47.847558 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:47.847523 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" event={"ID":"f89be472-d3aa-44a5-8d22-068bdea37f86","Type":"ContainerStarted","Data":"30458ce54346d68297192df114cbf959171bef404f7297ddfce5c29b55ac6fc8"} Jul 28 18:03:48.855289 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:48.855254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" event={"ID":"f89be472-d3aa-44a5-8d22-068bdea37f86","Type":"ContainerStarted","Data":"95076be1c681332cd7e9a179ef9c34797081eb9867df454101cf4e3ecefeba1b"} Jul 28 18:03:49.860974 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:49.860940 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" event={"ID":"f89be472-d3aa-44a5-8d22-068bdea37f86","Type":"ContainerStarted","Data":"f4e30fda99a697ae1985cdb9a507c5d74258eb78ed0a0d55f647257cc574f5d6"} Jul 28 18:03:49.887555 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:03:49.887503 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79bd4b8bff-7vqc6" podStartSLOduration=2.037437346 podStartE2EDuration="4.887486225s" totalImagesPullingTime="2.850048879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:49.886225329 +0000 UTC m=+174.499509758" watchObservedRunningTime="2026-07-28 18:03:49.887486225 +0000 UTC m=+174.500770634" Jul 28 18:05:52.438524 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.438491 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc"] Jul 28 18:05:52.441600 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.441581 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.444679 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.444650 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 18:05:52.444979 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.444963 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 18:05:52.445908 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.445892 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-87w5r\"" Jul 28 18:05:52.455792 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.455770 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc"] Jul 28 18:05:52.460471 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.460450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.460572 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.460478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.460634 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.460573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4s4r\" (UniqueName: \"kubernetes.io/projected/7695f428-eb20-495c-b72c-b121d12bab0c-kube-api-access-m4s4r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.561640 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.561605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4s4r\" (UniqueName: \"kubernetes.io/projected/7695f428-eb20-495c-b72c-b121d12bab0c-kube-api-access-m4s4r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.561794 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.561767 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.561844 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.561803 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.562152 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.562135 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.562191 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.562149 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.571600 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.571574 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4s4r\" (UniqueName: \"kubernetes.io/projected/7695f428-eb20-495c-b72c-b121d12bab0c-kube-api-access-m4s4r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.751447 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.751374 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:05:52.874867 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:52.874666 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc"] Jul 28 18:05:52.877636 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:05:52.877606 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7695f428_eb20_495c_b72c_b121d12bab0c.slice/crio-075e1859ccdef6bbea827be05c4d7aec8b0420f684b04eb4bca17d6d25d3221a WatchSource:0}: Error finding container 075e1859ccdef6bbea827be05c4d7aec8b0420f684b04eb4bca17d6d25d3221a: Status 404 returned error can't find the container with id 075e1859ccdef6bbea827be05c4d7aec8b0420f684b04eb4bca17d6d25d3221a Jul 28 18:05:53.336783 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:53.336743 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" event={"ID":"7695f428-eb20-495c-b72c-b121d12bab0c","Type":"ContainerStarted","Data":"075e1859ccdef6bbea827be05c4d7aec8b0420f684b04eb4bca17d6d25d3221a"} Jul 28 18:05:55.799308 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:55.799278 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:05:55.804089 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:55.804062 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 18:05:55.807535 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:55.807509 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:05:59.365319 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:59.365282 2569 generic.go:356] "Generic (PLEG): container finished" podID="7695f428-eb20-495c-b72c-b121d12bab0c" containerID="e21ade609c6034ff9840e273be8acb3d436c78cb9c62e92fcab1535d3b5027b4" exitCode=0 Jul 28 18:05:59.365767 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:59.365327 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" event={"ID":"7695f428-eb20-495c-b72c-b121d12bab0c","Type":"ContainerDied","Data":"e21ade609c6034ff9840e273be8acb3d436c78cb9c62e92fcab1535d3b5027b4"} Jul 28 18:05:59.366456 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:05:59.366437 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:06:01.376017 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:01.375926 2569 generic.go:356] "Generic (PLEG): container finished" podID="7695f428-eb20-495c-b72c-b121d12bab0c" containerID="947f6d4a9ee8891ec37b1dfe96206e6e1ef1f493c798b30f9090816d13e0b810" exitCode=0 Jul 28 18:06:01.376400 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:01.376014 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" event={"ID":"7695f428-eb20-495c-b72c-b121d12bab0c","Type":"ContainerDied","Data":"947f6d4a9ee8891ec37b1dfe96206e6e1ef1f493c798b30f9090816d13e0b810"} Jul 28 18:06:08.406693 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:08.406655 2569 generic.go:356] "Generic (PLEG): container finished" podID="7695f428-eb20-495c-b72c-b121d12bab0c" containerID="ac4a4e6fdcc0f1478608cea923fff3a0ffeaa739d82c10239514bf7cdbc12975" exitCode=0 Jul 28 18:06:08.407067 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:08.406711 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" event={"ID":"7695f428-eb20-495c-b72c-b121d12bab0c","Type":"ContainerDied","Data":"ac4a4e6fdcc0f1478608cea923fff3a0ffeaa739d82c10239514bf7cdbc12975"} Jul 28 18:06:09.534080 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.534050 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:06:09.615652 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.615618 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-util\") pod \"7695f428-eb20-495c-b72c-b121d12bab0c\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " Jul 28 18:06:09.615805 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.615705 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4s4r\" (UniqueName: \"kubernetes.io/projected/7695f428-eb20-495c-b72c-b121d12bab0c-kube-api-access-m4s4r\") pod \"7695f428-eb20-495c-b72c-b121d12bab0c\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " Jul 28 18:06:09.615805 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.615745 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-bundle\") pod \"7695f428-eb20-495c-b72c-b121d12bab0c\" (UID: \"7695f428-eb20-495c-b72c-b121d12bab0c\") " Jul 28 18:06:09.616396 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.616370 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-bundle" (OuterVolumeSpecName: "bundle") pod "7695f428-eb20-495c-b72c-b121d12bab0c" (UID: "7695f428-eb20-495c-b72c-b121d12bab0c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:09.617825 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.617805 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7695f428-eb20-495c-b72c-b121d12bab0c-kube-api-access-m4s4r" (OuterVolumeSpecName: "kube-api-access-m4s4r") pod "7695f428-eb20-495c-b72c-b121d12bab0c" (UID: "7695f428-eb20-495c-b72c-b121d12bab0c"). InnerVolumeSpecName "kube-api-access-m4s4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:06:09.619660 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.619635 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-util" (OuterVolumeSpecName: "util") pod "7695f428-eb20-495c-b72c-b121d12bab0c" (UID: "7695f428-eb20-495c-b72c-b121d12bab0c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:09.716579 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.716490 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4s4r\" (UniqueName: \"kubernetes.io/projected/7695f428-eb20-495c-b72c-b121d12bab0c-kube-api-access-m4s4r\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:06:09.716579 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.716527 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:06:09.716579 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:09.716538 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7695f428-eb20-495c-b72c-b121d12bab0c-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:06:10.415849 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:10.415816 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" event={"ID":"7695f428-eb20-495c-b72c-b121d12bab0c","Type":"ContainerDied","Data":"075e1859ccdef6bbea827be05c4d7aec8b0420f684b04eb4bca17d6d25d3221a"} Jul 28 18:06:10.415849 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:10.415856 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjfgc" Jul 28 18:06:10.416054 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:10.415867 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="075e1859ccdef6bbea827be05c4d7aec8b0420f684b04eb4bca17d6d25d3221a" Jul 28 18:06:15.803471 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.803437 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7"] Jul 28 18:06:15.803835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.803740 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7695f428-eb20-495c-b72c-b121d12bab0c" containerName="extract" Jul 28 18:06:15.803835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.803751 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7695f428-eb20-495c-b72c-b121d12bab0c" containerName="extract" Jul 28 18:06:15.803835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.803764 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7695f428-eb20-495c-b72c-b121d12bab0c" containerName="util" Jul 28 18:06:15.803835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.803770 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7695f428-eb20-495c-b72c-b121d12bab0c" containerName="util" Jul 28 18:06:15.803835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.803776 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7695f428-eb20-495c-b72c-b121d12bab0c" containerName="pull" Jul 28 18:06:15.803835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.803781 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7695f428-eb20-495c-b72c-b121d12bab0c" containerName="pull" Jul 28 18:06:15.803835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.803832 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7695f428-eb20-495c-b72c-b121d12bab0c" containerName="extract" Jul 28 18:06:15.840500 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.840466 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7"] Jul 28 18:06:15.840652 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.840580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" Jul 28 18:06:15.844202 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.844178 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 18:06:15.844359 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.844179 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mwcp7\"" Jul 28 18:06:15.844359 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.844261 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 18:06:15.970083 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.970046 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvh7\" (UniqueName: \"kubernetes.io/projected/06923a8f-6ba6-4618-9ecf-691f52fe603a-kube-api-access-cdvh7\") pod \"cert-manager-operator-controller-manager-55c444bf86-6tsv7\" (UID: \"06923a8f-6ba6-4618-9ecf-691f52fe603a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" Jul 28 18:06:15.970302 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:15.970195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06923a8f-6ba6-4618-9ecf-691f52fe603a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6tsv7\" (UID: \"06923a8f-6ba6-4618-9ecf-691f52fe603a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" Jul 28 18:06:16.071364 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:16.071282 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06923a8f-6ba6-4618-9ecf-691f52fe603a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6tsv7\" (UID: \"06923a8f-6ba6-4618-9ecf-691f52fe603a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" Jul 28 18:06:16.071364 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:16.071332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvh7\" (UniqueName: \"kubernetes.io/projected/06923a8f-6ba6-4618-9ecf-691f52fe603a-kube-api-access-cdvh7\") pod \"cert-manager-operator-controller-manager-55c444bf86-6tsv7\" (UID: \"06923a8f-6ba6-4618-9ecf-691f52fe603a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" Jul 28 18:06:16.071731 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:16.071707 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06923a8f-6ba6-4618-9ecf-691f52fe603a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6tsv7\" (UID: \"06923a8f-6ba6-4618-9ecf-691f52fe603a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" Jul 28 18:06:16.080928 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:16.080898 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvh7\" (UniqueName: \"kubernetes.io/projected/06923a8f-6ba6-4618-9ecf-691f52fe603a-kube-api-access-cdvh7\") pod \"cert-manager-operator-controller-manager-55c444bf86-6tsv7\" (UID: \"06923a8f-6ba6-4618-9ecf-691f52fe603a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" Jul 28 18:06:16.149871 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:16.149838 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" Jul 28 18:06:16.346788 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:16.346762 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7"] Jul 28 18:06:16.347739 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:06:16.347713 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06923a8f_6ba6_4618_9ecf_691f52fe603a.slice/crio-c845ce627aa2a72192401568ae6e9f40c2adcdc3fd30950ca62a8b4221267ec4 WatchSource:0}: Error finding container c845ce627aa2a72192401568ae6e9f40c2adcdc3fd30950ca62a8b4221267ec4: Status 404 returned error can't find the container with id c845ce627aa2a72192401568ae6e9f40c2adcdc3fd30950ca62a8b4221267ec4 Jul 28 18:06:16.442890 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:16.442856 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" event={"ID":"06923a8f-6ba6-4618-9ecf-691f52fe603a","Type":"ContainerStarted","Data":"c845ce627aa2a72192401568ae6e9f40c2adcdc3fd30950ca62a8b4221267ec4"} Jul 28 18:06:19.461876 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:19.461834 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" event={"ID":"06923a8f-6ba6-4618-9ecf-691f52fe603a","Type":"ContainerStarted","Data":"8ef9f43811ac8ca93573498be086e7bd5275c053b85929fb36e1c3601387b3b8"} Jul 28 18:06:19.489803 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:19.489733 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6tsv7" podStartSLOduration=2.455542249 podStartE2EDuration="4.489717795s" totalImagesPullingTime="2.034175546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:19.489497239 +0000 UTC m=+324.102781649" watchObservedRunningTime="2026-07-28 18:06:19.489717795 +0000 UTC m=+324.103002205" Jul 28 18:06:21.848641 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:21.848609 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xh5w7"] Jul 28 18:06:21.851024 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:21.850999 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:21.855789 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:21.855765 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 18:06:21.856053 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:21.856037 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 18:06:21.856826 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:21.856806 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-htpdb\"" Jul 28 18:06:21.866797 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:21.866772 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xh5w7"] Jul 28 18:06:21.922653 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:21.922618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab0a596b-8c1b-400b-8562-9250e157911b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xh5w7\" (UID: \"ab0a596b-8c1b-400b-8562-9250e157911b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:21.922836 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:21.922669 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxrgt\" (UniqueName: \"kubernetes.io/projected/ab0a596b-8c1b-400b-8562-9250e157911b-kube-api-access-xxrgt\") pod \"cert-manager-webhook-6486ffd997-xh5w7\" (UID: \"ab0a596b-8c1b-400b-8562-9250e157911b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:22.023652 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:22.023614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxrgt\" (UniqueName: \"kubernetes.io/projected/ab0a596b-8c1b-400b-8562-9250e157911b-kube-api-access-xxrgt\") pod \"cert-manager-webhook-6486ffd997-xh5w7\" (UID: \"ab0a596b-8c1b-400b-8562-9250e157911b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:22.023841 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:22.023711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab0a596b-8c1b-400b-8562-9250e157911b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xh5w7\" (UID: \"ab0a596b-8c1b-400b-8562-9250e157911b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:22.036818 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:22.036791 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab0a596b-8c1b-400b-8562-9250e157911b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xh5w7\" (UID: \"ab0a596b-8c1b-400b-8562-9250e157911b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:22.037086 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:22.037057 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxrgt\" (UniqueName: \"kubernetes.io/projected/ab0a596b-8c1b-400b-8562-9250e157911b-kube-api-access-xxrgt\") pod \"cert-manager-webhook-6486ffd997-xh5w7\" (UID: \"ab0a596b-8c1b-400b-8562-9250e157911b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:22.161006 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:22.160920 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:22.286454 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:22.286430 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xh5w7"] Jul 28 18:06:22.288644 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:06:22.288619 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab0a596b_8c1b_400b_8562_9250e157911b.slice/crio-fa506a4f9ef310c51227aee95f342e2cb8f2f6cee41671318fc7ceea4a644b38 WatchSource:0}: Error finding container fa506a4f9ef310c51227aee95f342e2cb8f2f6cee41671318fc7ceea4a644b38: Status 404 returned error can't find the container with id fa506a4f9ef310c51227aee95f342e2cb8f2f6cee41671318fc7ceea4a644b38 Jul 28 18:06:22.475706 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:22.475621 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" event={"ID":"ab0a596b-8c1b-400b-8562-9250e157911b","Type":"ContainerStarted","Data":"fa506a4f9ef310c51227aee95f342e2cb8f2f6cee41671318fc7ceea4a644b38"} Jul 28 18:06:23.033736 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.033695 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ck45z"] Jul 28 18:06:23.036087 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.036066 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" Jul 28 18:06:23.040257 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.039267 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x5jxb\"" Jul 28 18:06:23.047276 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.047250 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ck45z"] Jul 28 18:06:23.131792 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.131754 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41cd2d13-9cbc-483a-9fbd-d137967944a2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ck45z\" (UID: \"41cd2d13-9cbc-483a-9fbd-d137967944a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" Jul 28 18:06:23.131978 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.131865 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lblxx\" (UniqueName: \"kubernetes.io/projected/41cd2d13-9cbc-483a-9fbd-d137967944a2-kube-api-access-lblxx\") pod \"cert-manager-cainjector-559cdd8c87-ck45z\" (UID: \"41cd2d13-9cbc-483a-9fbd-d137967944a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" Jul 28 18:06:23.232475 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.232441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41cd2d13-9cbc-483a-9fbd-d137967944a2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ck45z\" (UID: \"41cd2d13-9cbc-483a-9fbd-d137967944a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" Jul 28 18:06:23.232630 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.232500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lblxx\" (UniqueName: \"kubernetes.io/projected/41cd2d13-9cbc-483a-9fbd-d137967944a2-kube-api-access-lblxx\") pod \"cert-manager-cainjector-559cdd8c87-ck45z\" (UID: \"41cd2d13-9cbc-483a-9fbd-d137967944a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" Jul 28 18:06:23.243949 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.243891 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41cd2d13-9cbc-483a-9fbd-d137967944a2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ck45z\" (UID: \"41cd2d13-9cbc-483a-9fbd-d137967944a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" Jul 28 18:06:23.244135 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.244112 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lblxx\" (UniqueName: \"kubernetes.io/projected/41cd2d13-9cbc-483a-9fbd-d137967944a2-kube-api-access-lblxx\") pod \"cert-manager-cainjector-559cdd8c87-ck45z\" (UID: \"41cd2d13-9cbc-483a-9fbd-d137967944a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" Jul 28 18:06:23.349286 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.349252 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" Jul 28 18:06:23.500200 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:23.500142 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ck45z"] Jul 28 18:06:23.502189 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:06:23.502163 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41cd2d13_9cbc_483a_9fbd_d137967944a2.slice/crio-79118244d970eb2f2faee2a82cbc43a795433bfeaedbcb232fcbabc53c531b9f WatchSource:0}: Error finding container 79118244d970eb2f2faee2a82cbc43a795433bfeaedbcb232fcbabc53c531b9f: Status 404 returned error can't find the container with id 79118244d970eb2f2faee2a82cbc43a795433bfeaedbcb232fcbabc53c531b9f Jul 28 18:06:24.487642 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:24.487599 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" event={"ID":"41cd2d13-9cbc-483a-9fbd-d137967944a2","Type":"ContainerStarted","Data":"79118244d970eb2f2faee2a82cbc43a795433bfeaedbcb232fcbabc53c531b9f"} Jul 28 18:06:25.493638 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:25.493594 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" event={"ID":"ab0a596b-8c1b-400b-8562-9250e157911b","Type":"ContainerStarted","Data":"3bf0817cf625c1e889284396b8148f91200c5ed3be7a3b5b539ee4ea2a7e6900"} Jul 28 18:06:25.494110 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:25.493655 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:25.495071 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:25.495049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" event={"ID":"41cd2d13-9cbc-483a-9fbd-d137967944a2","Type":"ContainerStarted","Data":"fa3003319e173131912bfed92f4cec82984a72351847a0212ab33faef285fda9"} Jul 28 18:06:25.523422 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:25.523363 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" podStartSLOduration=2.032598031 podStartE2EDuration="4.523343094s" totalImagesPullingTime="2.490745063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:25.522088463 +0000 UTC m=+330.135372852" watchObservedRunningTime="2026-07-28 18:06:25.523343094 +0000 UTC m=+330.136627504" Jul 28 18:06:25.541381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:25.541326 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ck45z" podStartSLOduration=1.26452653 podStartE2EDuration="2.541307718s" totalImagesPullingTime="1.276781188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:25.5409933 +0000 UTC m=+330.154277732" watchObservedRunningTime="2026-07-28 18:06:25.541307718 +0000 UTC m=+330.154592129" Jul 28 18:06:31.501889 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:31.501857 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xh5w7" Jul 28 18:06:51.242962 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.242884 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh"] Jul 28 18:06:51.247599 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.247582 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.250696 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.250674 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 18:06:51.250881 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.250736 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-87w5r\"" Jul 28 18:06:51.250881 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.250740 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 18:06:51.255087 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.254692 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh"] Jul 28 18:06:51.270240 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.270207 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.270339 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.270305 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpm4\" (UniqueName: \"kubernetes.io/projected/067fcc2c-8d95-4aae-947b-d1347760b3e6-kube-api-access-qfpm4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.270379 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.270344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.371556 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.371520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpm4\" (UniqueName: \"kubernetes.io/projected/067fcc2c-8d95-4aae-947b-d1347760b3e6-kube-api-access-qfpm4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.371556 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.371558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.371768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.371743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.371894 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.371880 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.372050 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.372034 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.380223 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.380197 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpm4\" (UniqueName: \"kubernetes.io/projected/067fcc2c-8d95-4aae-947b-d1347760b3e6-kube-api-access-qfpm4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.557929 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.557850 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:51.691026 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:51.691000 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh"] Jul 28 18:06:51.693688 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:06:51.693664 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067fcc2c_8d95_4aae_947b_d1347760b3e6.slice/crio-51fa50dac9af6a8fa4a8bf9b5b2730bda98a6bfb433997f0f30d63a863152873 WatchSource:0}: Error finding container 51fa50dac9af6a8fa4a8bf9b5b2730bda98a6bfb433997f0f30d63a863152873: Status 404 returned error can't find the container with id 51fa50dac9af6a8fa4a8bf9b5b2730bda98a6bfb433997f0f30d63a863152873 Jul 28 18:06:52.613620 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:52.613588 2569 generic.go:356] "Generic (PLEG): container finished" podID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerID="4ed60a11a9ca84a4b3dc589915deef573613b0788d685837e661c509cf89b5d2" exitCode=0 Jul 28 18:06:52.613984 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:52.613672 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" event={"ID":"067fcc2c-8d95-4aae-947b-d1347760b3e6","Type":"ContainerDied","Data":"4ed60a11a9ca84a4b3dc589915deef573613b0788d685837e661c509cf89b5d2"} Jul 28 18:06:52.613984 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:52.613720 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" event={"ID":"067fcc2c-8d95-4aae-947b-d1347760b3e6","Type":"ContainerStarted","Data":"51fa50dac9af6a8fa4a8bf9b5b2730bda98a6bfb433997f0f30d63a863152873"} Jul 28 18:06:53.620412 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:53.620376 2569 generic.go:356] "Generic (PLEG): container finished" podID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerID="8772a9f5093d9f2e699d849ea2f0a86871c910392f209ee427b587ab4747b46f" exitCode=0 Jul 28 18:06:53.620826 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:53.620470 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" event={"ID":"067fcc2c-8d95-4aae-947b-d1347760b3e6","Type":"ContainerDied","Data":"8772a9f5093d9f2e699d849ea2f0a86871c910392f209ee427b587ab4747b46f"} Jul 28 18:06:54.627039 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:54.626997 2569 generic.go:356] "Generic (PLEG): container finished" podID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerID="984b1c2fe7e80c2d0ff289edca3a1e3d2d591cc1b2dcbc503a2846ac777d816c" exitCode=0 Jul 28 18:06:54.627426 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:54.627089 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" event={"ID":"067fcc2c-8d95-4aae-947b-d1347760b3e6","Type":"ContainerDied","Data":"984b1c2fe7e80c2d0ff289edca3a1e3d2d591cc1b2dcbc503a2846ac777d816c"} Jul 28 18:06:55.754030 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.754008 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:55.807021 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.806997 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-util\") pod \"067fcc2c-8d95-4aae-947b-d1347760b3e6\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " Jul 28 18:06:55.807158 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.807060 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfpm4\" (UniqueName: \"kubernetes.io/projected/067fcc2c-8d95-4aae-947b-d1347760b3e6-kube-api-access-qfpm4\") pod \"067fcc2c-8d95-4aae-947b-d1347760b3e6\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " Jul 28 18:06:55.807158 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.807100 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-bundle\") pod \"067fcc2c-8d95-4aae-947b-d1347760b3e6\" (UID: \"067fcc2c-8d95-4aae-947b-d1347760b3e6\") " Jul 28 18:06:55.807790 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.807767 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-bundle" (OuterVolumeSpecName: "bundle") pod "067fcc2c-8d95-4aae-947b-d1347760b3e6" (UID: "067fcc2c-8d95-4aae-947b-d1347760b3e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:55.809100 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.809076 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067fcc2c-8d95-4aae-947b-d1347760b3e6-kube-api-access-qfpm4" (OuterVolumeSpecName: "kube-api-access-qfpm4") pod "067fcc2c-8d95-4aae-947b-d1347760b3e6" (UID: "067fcc2c-8d95-4aae-947b-d1347760b3e6"). InnerVolumeSpecName "kube-api-access-qfpm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:06:55.812996 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.812954 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-util" (OuterVolumeSpecName: "util") pod "067fcc2c-8d95-4aae-947b-d1347760b3e6" (UID: "067fcc2c-8d95-4aae-947b-d1347760b3e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:55.907860 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.907783 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:06:55.907860 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.907809 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfpm4\" (UniqueName: \"kubernetes.io/projected/067fcc2c-8d95-4aae-947b-d1347760b3e6-kube-api-access-qfpm4\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:06:55.907860 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:55.907821 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/067fcc2c-8d95-4aae-947b-d1347760b3e6-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:06:56.637118 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:56.637080 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" event={"ID":"067fcc2c-8d95-4aae-947b-d1347760b3e6","Type":"ContainerDied","Data":"51fa50dac9af6a8fa4a8bf9b5b2730bda98a6bfb433997f0f30d63a863152873"} Jul 28 18:06:56.637315 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:56.637128 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc8hhh" Jul 28 18:06:56.637315 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:06:56.637140 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51fa50dac9af6a8fa4a8bf9b5b2730bda98a6bfb433997f0f30d63a863152873" Jul 28 18:07:08.338531 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.338489 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l"] Jul 28 18:07:08.339109 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.338950 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerName="extract" Jul 28 18:07:08.339109 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.338969 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerName="extract" Jul 28 18:07:08.339109 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.338986 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerName="pull" Jul 28 18:07:08.339109 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.339013 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerName="pull" Jul 28 18:07:08.339109 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.339027 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerName="util" Jul 28 18:07:08.339109 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.339036 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerName="util" Jul 28 18:07:08.339450 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.339181 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="067fcc2c-8d95-4aae-947b-d1347760b3e6" containerName="extract" Jul 28 18:07:08.342276 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.342255 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.345078 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.345057 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 18:07:08.345290 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.345271 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 18:07:08.345622 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.345601 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sz7wn\"" Jul 28 18:07:08.345679 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.345631 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 18:07:08.345723 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.345633 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 18:07:08.365001 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.364964 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l"] Jul 28 18:07:08.422022 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.421986 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfsx7\" (UniqueName: \"kubernetes.io/projected/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-kube-api-access-rfsx7\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.422200 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.422053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-tmp\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.422200 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.422092 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.422200 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.422132 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-manifests\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.422200 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.422163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.523081 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.523042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-tmp\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.523081 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.523086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.523369 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.523118 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-manifests\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.523369 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.523141 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.523369 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.523162 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfsx7\" (UniqueName: \"kubernetes.io/projected/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-kube-api-access-rfsx7\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.523601 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.523576 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-manifests\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.525471 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.525437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-tmp\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.525606 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.525584 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.525661 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.525585 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.533970 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.533945 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfsx7\" (UniqueName: \"kubernetes.io/projected/235a57a7-ce35-46fb-95b7-1d4331b2c1e0-kube-api-access-rfsx7\") pod \"opendatahub-operator-controller-manager-7795fd7c74-nt97l\" (UID: \"235a57a7-ce35-46fb-95b7-1d4331b2c1e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.653076 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.653035 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:08.786670 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:08.786634 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l"] Jul 28 18:07:08.789005 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:07:08.788979 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235a57a7_ce35_46fb_95b7_1d4331b2c1e0.slice/crio-fdb928cba1c7ab6d93671d97109f603b9b9107ce624bf4aae1229d0bb09285b6 WatchSource:0}: Error finding container fdb928cba1c7ab6d93671d97109f603b9b9107ce624bf4aae1229d0bb09285b6: Status 404 returned error can't find the container with id fdb928cba1c7ab6d93671d97109f603b9b9107ce624bf4aae1229d0bb09285b6 Jul 28 18:07:09.697353 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:09.697265 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" event={"ID":"235a57a7-ce35-46fb-95b7-1d4331b2c1e0","Type":"ContainerStarted","Data":"fdb928cba1c7ab6d93671d97109f603b9b9107ce624bf4aae1229d0bb09285b6"} Jul 28 18:07:12.713894 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:12.713856 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" event={"ID":"235a57a7-ce35-46fb-95b7-1d4331b2c1e0","Type":"ContainerStarted","Data":"50af9b6a71cc1fd20cde3ca375fdb438ee068fec115a8bf162e9da982c0244f1"} Jul 28 18:07:13.720170 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:13.720133 2569 generic.go:356] "Generic (PLEG): container finished" podID="235a57a7-ce35-46fb-95b7-1d4331b2c1e0" containerID="50af9b6a71cc1fd20cde3ca375fdb438ee068fec115a8bf162e9da982c0244f1" exitCode=0 Jul 28 18:07:13.720561 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:13.720198 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" event={"ID":"235a57a7-ce35-46fb-95b7-1d4331b2c1e0","Type":"ContainerDied","Data":"50af9b6a71cc1fd20cde3ca375fdb438ee068fec115a8bf162e9da982c0244f1"} Jul 28 18:07:14.726575 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:14.726542 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" event={"ID":"235a57a7-ce35-46fb-95b7-1d4331b2c1e0","Type":"ContainerStarted","Data":"c8e258eeb81405760331a2e1bb38375f2949242772b27f59397a98da51e962fd"} Jul 28 18:07:14.727044 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:14.726684 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:14.752167 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:14.752105 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" podStartSLOduration=2.165221634 podStartE2EDuration="6.752086913s" totalImagesPullingTime="3.551175129s" totalInitContainerRuntime="1.03569015s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:14.749745004 +0000 UTC m=+379.363029422" watchObservedRunningTime="2026-07-28 18:07:14.752086913 +0000 UTC m=+379.365371322" Jul 28 18:07:25.734205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:25.734174 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7795fd7c74-nt97l" Jul 28 18:07:28.401953 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.401915 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kft7m"] Jul 28 18:07:28.410090 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.410047 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:28.413221 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.413193 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 28 18:07:28.413411 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.413197 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-n6ft6\"" Jul 28 18:07:28.419457 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.419336 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kft7m"] Jul 28 18:07:28.605142 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.605111 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43d52037-1291-4aa3-977d-dfa65ca62b05-cert\") pod \"odh-model-controller-7998c67dc4-kft7m\" (UID: \"43d52037-1291-4aa3-977d-dfa65ca62b05\") " pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:28.605339 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.605213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5h2g\" (UniqueName: \"kubernetes.io/projected/43d52037-1291-4aa3-977d-dfa65ca62b05-kube-api-access-d5h2g\") pod \"odh-model-controller-7998c67dc4-kft7m\" (UID: \"43d52037-1291-4aa3-977d-dfa65ca62b05\") " pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:28.705860 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.705779 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43d52037-1291-4aa3-977d-dfa65ca62b05-cert\") pod \"odh-model-controller-7998c67dc4-kft7m\" (UID: \"43d52037-1291-4aa3-977d-dfa65ca62b05\") " pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:28.706016 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.705902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5h2g\" (UniqueName: \"kubernetes.io/projected/43d52037-1291-4aa3-977d-dfa65ca62b05-kube-api-access-d5h2g\") pod \"odh-model-controller-7998c67dc4-kft7m\" (UID: \"43d52037-1291-4aa3-977d-dfa65ca62b05\") " pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:28.708151 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.708127 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43d52037-1291-4aa3-977d-dfa65ca62b05-cert\") pod \"odh-model-controller-7998c67dc4-kft7m\" (UID: \"43d52037-1291-4aa3-977d-dfa65ca62b05\") " pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:28.717831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.717803 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5h2g\" (UniqueName: \"kubernetes.io/projected/43d52037-1291-4aa3-977d-dfa65ca62b05-kube-api-access-d5h2g\") pod \"odh-model-controller-7998c67dc4-kft7m\" (UID: \"43d52037-1291-4aa3-977d-dfa65ca62b05\") " pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:28.723711 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.723689 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:28.860275 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:28.860247 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kft7m"] Jul 28 18:07:28.861917 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:07:28.861890 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d52037_1291_4aa3_977d_dfa65ca62b05.slice/crio-35b4a9f896237e709dc4278df47618fa384f4d6b609566b6e8391a8cc0765206 WatchSource:0}: Error finding container 35b4a9f896237e709dc4278df47618fa384f4d6b609566b6e8391a8cc0765206: Status 404 returned error can't find the container with id 35b4a9f896237e709dc4278df47618fa384f4d6b609566b6e8391a8cc0765206 Jul 28 18:07:29.103740 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.103708 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vq8ws"] Jul 28 18:07:29.106616 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.106597 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:29.109319 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.109299 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 28 18:07:29.119605 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.119577 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vq8ws"] Jul 28 18:07:29.210354 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.210314 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4h74\" (UniqueName: \"kubernetes.io/projected/7192e8d6-283a-4eae-b732-982758f8c802-kube-api-access-p4h74\") pod \"model-serving-api-54d89bd5fc-vq8ws\" (UID: \"7192e8d6-283a-4eae-b732-982758f8c802\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:29.210601 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.210382 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7192e8d6-283a-4eae-b732-982758f8c802-tls-certs\") pod \"model-serving-api-54d89bd5fc-vq8ws\" (UID: \"7192e8d6-283a-4eae-b732-982758f8c802\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:29.311426 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.311391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4h74\" (UniqueName: \"kubernetes.io/projected/7192e8d6-283a-4eae-b732-982758f8c802-kube-api-access-p4h74\") pod \"model-serving-api-54d89bd5fc-vq8ws\" (UID: \"7192e8d6-283a-4eae-b732-982758f8c802\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:29.311614 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.311440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7192e8d6-283a-4eae-b732-982758f8c802-tls-certs\") pod \"model-serving-api-54d89bd5fc-vq8ws\" (UID: \"7192e8d6-283a-4eae-b732-982758f8c802\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:29.313868 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.313845 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7192e8d6-283a-4eae-b732-982758f8c802-tls-certs\") pod \"model-serving-api-54d89bd5fc-vq8ws\" (UID: \"7192e8d6-283a-4eae-b732-982758f8c802\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:29.320591 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.320572 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4h74\" (UniqueName: \"kubernetes.io/projected/7192e8d6-283a-4eae-b732-982758f8c802-kube-api-access-p4h74\") pod \"model-serving-api-54d89bd5fc-vq8ws\" (UID: \"7192e8d6-283a-4eae-b732-982758f8c802\") " pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:29.419324 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.419222 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:29.546094 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.546062 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vq8ws"] Jul 28 18:07:29.547621 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:07:29.547595 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7192e8d6_283a_4eae_b732_982758f8c802.slice/crio-4ebac38dd63c2ba4f7ae8251ae5073bb126651832da95a8ba563441250d0b483 WatchSource:0}: Error finding container 4ebac38dd63c2ba4f7ae8251ae5073bb126651832da95a8ba563441250d0b483: Status 404 returned error can't find the container with id 4ebac38dd63c2ba4f7ae8251ae5073bb126651832da95a8ba563441250d0b483 Jul 28 18:07:29.801451 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.801340 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" event={"ID":"7192e8d6-283a-4eae-b732-982758f8c802","Type":"ContainerStarted","Data":"4ebac38dd63c2ba4f7ae8251ae5073bb126651832da95a8ba563441250d0b483"} Jul 28 18:07:29.802982 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:29.802952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" event={"ID":"43d52037-1291-4aa3-977d-dfa65ca62b05","Type":"ContainerStarted","Data":"35b4a9f896237e709dc4278df47618fa384f4d6b609566b6e8391a8cc0765206"} Jul 28 18:07:32.819981 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:32.819940 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" event={"ID":"43d52037-1291-4aa3-977d-dfa65ca62b05","Type":"ContainerStarted","Data":"05d7637337c27df7c45e52e5976369a31227e1d6ed07e0152d98bb9c783ccd31"} Jul 28 18:07:32.820467 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:32.820004 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:32.821387 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:32.821333 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" event={"ID":"7192e8d6-283a-4eae-b732-982758f8c802","Type":"ContainerStarted","Data":"d4d835112062916bc74e8b989fc1258103e69643c9d8fdd804dc22f511299b51"} Jul 28 18:07:32.821387 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:32.821386 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:32.839667 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:32.839622 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" podStartSLOduration=1.549842506 podStartE2EDuration="4.839609413s" totalImagesPullingTime="3.289766907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:32.837774791 +0000 UTC m=+397.451059204" watchObservedRunningTime="2026-07-28 18:07:32.839609413 +0000 UTC m=+397.452893822" Jul 28 18:07:32.856315 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:32.856261 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" podStartSLOduration=1.25536928 podStartE2EDuration="3.856225749s" totalImagesPullingTime="2.600856469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:32.854366793 +0000 UTC m=+397.467651203" watchObservedRunningTime="2026-07-28 18:07:32.856225749 +0000 UTC m=+397.469510159" Jul 28 18:07:37.944462 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.943993 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh"] Jul 28 18:07:37.948178 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.948155 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:37.950959 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.950934 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 28 18:07:37.951255 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.951220 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-s5s7r\"" Jul 28 18:07:37.963504 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.963476 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s"] Jul 28 18:07:37.967838 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.967816 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:37.968276 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.968252 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh"] Jul 28 18:07:37.971371 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.971347 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 28 18:07:37.971587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.971571 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vk6sw\"" Jul 28 18:07:37.982273 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.982251 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s"] Jul 28 18:07:37.989672 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.989649 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47h9\" (UniqueName: \"kubernetes.io/projected/fc15845f-7f33-4334-8531-6da0357fd1b3-kube-api-access-r47h9\") pod \"kserve-controller-manager-d6f4b65b8-xwkbh\" (UID: \"fc15845f-7f33-4334-8531-6da0357fd1b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:37.989889 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.989871 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwkbh\" (UID: \"fc15845f-7f33-4334-8531-6da0357fd1b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:37.990014 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.989991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjtn\" (UniqueName: \"kubernetes.io/projected/5505cefc-5b40-4812-be42-aa884e5e176c-kube-api-access-2qjtn\") pod \"llmisvc-controller-manager-777c6bd7d5-mnn5s\" (UID: \"5505cefc-5b40-4812-be42-aa884e5e176c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:37.990301 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:37.990278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5505cefc-5b40-4812-be42-aa884e5e176c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mnn5s\" (UID: \"5505cefc-5b40-4812-be42-aa884e5e176c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:38.091045 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.091012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47h9\" (UniqueName: \"kubernetes.io/projected/fc15845f-7f33-4334-8531-6da0357fd1b3-kube-api-access-r47h9\") pod \"kserve-controller-manager-d6f4b65b8-xwkbh\" (UID: \"fc15845f-7f33-4334-8531-6da0357fd1b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:38.091217 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.091059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwkbh\" (UID: \"fc15845f-7f33-4334-8531-6da0357fd1b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:38.091217 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.091080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjtn\" (UniqueName: \"kubernetes.io/projected/5505cefc-5b40-4812-be42-aa884e5e176c-kube-api-access-2qjtn\") pod \"llmisvc-controller-manager-777c6bd7d5-mnn5s\" (UID: \"5505cefc-5b40-4812-be42-aa884e5e176c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:38.091217 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.091103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5505cefc-5b40-4812-be42-aa884e5e176c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mnn5s\" (UID: \"5505cefc-5b40-4812-be42-aa884e5e176c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:38.091217 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:38.091187 2569 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 28 18:07:38.091461 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:38.091293 2569 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 28 18:07:38.091461 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:38.091300 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert podName:fc15845f-7f33-4334-8531-6da0357fd1b3 nodeName:}" failed. No retries permitted until 2026-07-28 18:07:38.591277306 +0000 UTC m=+403.204561697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert") pod "kserve-controller-manager-d6f4b65b8-xwkbh" (UID: "fc15845f-7f33-4334-8531-6da0357fd1b3") : secret "kserve-webhook-server-cert" not found Jul 28 18:07:38.091461 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:38.091377 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5505cefc-5b40-4812-be42-aa884e5e176c-cert podName:5505cefc-5b40-4812-be42-aa884e5e176c nodeName:}" failed. No retries permitted until 2026-07-28 18:07:38.591359329 +0000 UTC m=+403.204643725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5505cefc-5b40-4812-be42-aa884e5e176c-cert") pod "llmisvc-controller-manager-777c6bd7d5-mnn5s" (UID: "5505cefc-5b40-4812-be42-aa884e5e176c") : secret "llmisvc-webhook-server-cert" not found Jul 28 18:07:38.105901 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.105875 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47h9\" (UniqueName: \"kubernetes.io/projected/fc15845f-7f33-4334-8531-6da0357fd1b3-kube-api-access-r47h9\") pod \"kserve-controller-manager-d6f4b65b8-xwkbh\" (UID: \"fc15845f-7f33-4334-8531-6da0357fd1b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:38.105993 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.105906 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjtn\" (UniqueName: \"kubernetes.io/projected/5505cefc-5b40-4812-be42-aa884e5e176c-kube-api-access-2qjtn\") pod \"llmisvc-controller-manager-777c6bd7d5-mnn5s\" (UID: \"5505cefc-5b40-4812-be42-aa884e5e176c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:38.473871 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.473835 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck"] Jul 28 18:07:38.477381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.477363 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.481114 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.481063 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 28 18:07:38.481272 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.481171 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 28 18:07:38.481431 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.481415 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 18:07:38.481747 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.481726 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 18:07:38.494471 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.494450 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck"] Jul 28 18:07:38.596474 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.596440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwkbh\" (UID: \"fc15845f-7f33-4334-8531-6da0357fd1b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:38.596474 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.596489 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.596724 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.596540 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ac59740-707e-4a48-8dec-ad3a21a89be0-tmp\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.596724 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.596572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5505cefc-5b40-4812-be42-aa884e5e176c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mnn5s\" (UID: \"5505cefc-5b40-4812-be42-aa884e5e176c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:38.596724 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:38.596614 2569 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 28 18:07:38.596724 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.596679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc9mq\" (UniqueName: \"kubernetes.io/projected/4ac59740-707e-4a48-8dec-ad3a21a89be0-kube-api-access-zc9mq\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.596724 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:38.596687 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert podName:fc15845f-7f33-4334-8531-6da0357fd1b3 nodeName:}" failed. No retries permitted until 2026-07-28 18:07:39.596663863 +0000 UTC m=+404.209948253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert") pod "kserve-controller-manager-d6f4b65b8-xwkbh" (UID: "fc15845f-7f33-4334-8531-6da0357fd1b3") : secret "kserve-webhook-server-cert" not found Jul 28 18:07:38.599077 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.599044 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5505cefc-5b40-4812-be42-aa884e5e176c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mnn5s\" (UID: \"5505cefc-5b40-4812-be42-aa884e5e176c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:38.697524 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.697487 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc9mq\" (UniqueName: \"kubernetes.io/projected/4ac59740-707e-4a48-8dec-ad3a21a89be0-kube-api-access-zc9mq\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.697687 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.697584 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.697687 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.697606 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ac59740-707e-4a48-8dec-ad3a21a89be0-tmp\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.697763 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:38.697730 2569 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 28 18:07:38.697811 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:38.697799 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs podName:4ac59740-707e-4a48-8dec-ad3a21a89be0 nodeName:}" failed. No retries permitted until 2026-07-28 18:07:39.197782563 +0000 UTC m=+403.811066951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs") pod "kube-auth-proxy-79b97f775b-kg9ck" (UID: "4ac59740-707e-4a48-8dec-ad3a21a89be0") : secret "kube-auth-proxy-tls" not found Jul 28 18:07:38.699795 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.699771 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ac59740-707e-4a48-8dec-ad3a21a89be0-tmp\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.711659 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.711639 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc9mq\" (UniqueName: \"kubernetes.io/projected/4ac59740-707e-4a48-8dec-ad3a21a89be0-kube-api-access-zc9mq\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:38.879465 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:38.879425 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:39.016681 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:39.016660 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s"] Jul 28 18:07:39.018753 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:07:39.018727 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5505cefc_5b40_4812_be42_aa884e5e176c.slice/crio-8693ba175d8ea00c5d53cc78b4207dad57b315aad8ad11dac4b592193e54d721 WatchSource:0}: Error finding container 8693ba175d8ea00c5d53cc78b4207dad57b315aad8ad11dac4b592193e54d721: Status 404 returned error can't find the container with id 8693ba175d8ea00c5d53cc78b4207dad57b315aad8ad11dac4b592193e54d721 Jul 28 18:07:39.202557 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:39.202475 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:39.202721 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:39.202578 2569 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 28 18:07:39.202721 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:07:39.202631 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs podName:4ac59740-707e-4a48-8dec-ad3a21a89be0 nodeName:}" failed. No retries permitted until 2026-07-28 18:07:40.202617072 +0000 UTC m=+404.815901459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs") pod "kube-auth-proxy-79b97f775b-kg9ck" (UID: "4ac59740-707e-4a48-8dec-ad3a21a89be0") : secret "kube-auth-proxy-tls" not found Jul 28 18:07:39.605953 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:39.605919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwkbh\" (UID: \"fc15845f-7f33-4334-8531-6da0357fd1b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:39.608327 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:39.608298 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc15845f-7f33-4334-8531-6da0357fd1b3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwkbh\" (UID: \"fc15845f-7f33-4334-8531-6da0357fd1b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:39.761441 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:39.761400 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:39.860913 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:39.860833 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" event={"ID":"5505cefc-5b40-4812-be42-aa884e5e176c","Type":"ContainerStarted","Data":"8693ba175d8ea00c5d53cc78b4207dad57b315aad8ad11dac4b592193e54d721"} Jul 28 18:07:39.967093 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:39.967065 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh"] Jul 28 18:07:40.066638 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:07:40.066602 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc15845f_7f33_4334_8531_6da0357fd1b3.slice/crio-5be9e20760e630195358936a9bab26e2faa9f9b668f0520a681eb3f0fbb323a8 WatchSource:0}: Error finding container 5be9e20760e630195358936a9bab26e2faa9f9b668f0520a681eb3f0fbb323a8: Status 404 returned error can't find the container with id 5be9e20760e630195358936a9bab26e2faa9f9b668f0520a681eb3f0fbb323a8 Jul 28 18:07:40.215299 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.214626 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:40.223955 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.223726 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac59740-707e-4a48-8dec-ad3a21a89be0-tls-certs\") pod \"kube-auth-proxy-79b97f775b-kg9ck\" (UID: \"4ac59740-707e-4a48-8dec-ad3a21a89be0\") " pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:40.288059 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.288016 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" Jul 28 18:07:40.556076 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.552410 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj"] Jul 28 18:07:40.560600 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.560559 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.564600 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.564421 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 18:07:40.570483 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.570305 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 18:07:40.570620 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.570565 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 18:07:40.584107 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.583905 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 18:07:40.584583 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.584560 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7tfs7\"" Jul 28 18:07:40.624986 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.624961 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.625258 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.624998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6616f83-1226-42c4-8dc1-aa7b02b15589-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.625258 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.625042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.625258 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.625079 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgn54\" (UniqueName: \"kubernetes.io/projected/c6616f83-1226-42c4-8dc1-aa7b02b15589-kube-api-access-qgn54\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.625258 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.625115 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.625258 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.625146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.625258 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.625195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.634531 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.634195 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj"] Jul 28 18:07:40.686677 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.686652 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck"] Jul 28 18:07:40.688507 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:07:40.688469 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac59740_707e_4a48_8dec_ad3a21a89be0.slice/crio-4bf347d1398dc97de6a9988bc3273af9b001bc264a2a144bc8ec1f852000d949 WatchSource:0}: Error finding container 4bf347d1398dc97de6a9988bc3273af9b001bc264a2a144bc8ec1f852000d949: Status 404 returned error can't find the container with id 4bf347d1398dc97de6a9988bc3273af9b001bc264a2a144bc8ec1f852000d949 Jul 28 18:07:40.726292 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.726263 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.726421 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.726303 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.726421 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.726383 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.726421 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.726412 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6616f83-1226-42c4-8dc1-aa7b02b15589-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.726584 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.726554 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.726642 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.726597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgn54\" (UniqueName: \"kubernetes.io/projected/c6616f83-1226-42c4-8dc1-aa7b02b15589-kube-api-access-qgn54\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.726642 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.726628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.727277 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.727220 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.729033 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.728988 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.729278 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.729257 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6616f83-1226-42c4-8dc1-aa7b02b15589-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.729278 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.729274 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.729417 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.729332 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.735585 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.735544 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6616f83-1226-42c4-8dc1-aa7b02b15589-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.736152 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.736130 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgn54\" (UniqueName: \"kubernetes.io/projected/c6616f83-1226-42c4-8dc1-aa7b02b15589-kube-api-access-qgn54\") pod \"istiod-openshift-gateway-5677ccb49b-954rj\" (UID: \"c6616f83-1226-42c4-8dc1-aa7b02b15589\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:40.867141 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.867100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" event={"ID":"4ac59740-707e-4a48-8dec-ad3a21a89be0","Type":"ContainerStarted","Data":"4bf347d1398dc97de6a9988bc3273af9b001bc264a2a144bc8ec1f852000d949"} Jul 28 18:07:40.868184 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.868151 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" event={"ID":"fc15845f-7f33-4334-8531-6da0357fd1b3","Type":"ContainerStarted","Data":"5be9e20760e630195358936a9bab26e2faa9f9b668f0520a681eb3f0fbb323a8"} Jul 28 18:07:40.875421 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:40.875395 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:41.030847 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:41.030640 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj"] Jul 28 18:07:41.032623 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:07:41.032592 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6616f83_1226_42c4_8dc1_aa7b02b15589.slice/crio-f012dfcef479bf17bddd335a95f0bf2d070fc998282143b8981bd25195b37027 WatchSource:0}: Error finding container f012dfcef479bf17bddd335a95f0bf2d070fc998282143b8981bd25195b37027: Status 404 returned error can't find the container with id f012dfcef479bf17bddd335a95f0bf2d070fc998282143b8981bd25195b37027 Jul 28 18:07:41.876791 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:41.876741 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" event={"ID":"c6616f83-1226-42c4-8dc1-aa7b02b15589","Type":"ContainerStarted","Data":"f012dfcef479bf17bddd335a95f0bf2d070fc998282143b8981bd25195b37027"} Jul 28 18:07:43.828143 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:43.828106 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kft7m" Jul 28 18:07:43.831123 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:43.831101 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vq8ws" Jul 28 18:07:44.757788 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.757595 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 28 18:07:44.757788 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.757677 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 28 18:07:44.896950 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.896909 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" event={"ID":"c6616f83-1226-42c4-8dc1-aa7b02b15589","Type":"ContainerStarted","Data":"bae4c4ae05500013c0bdb8f8c36704119fcb790ab13fa9dce12378b67e4e6da7"} Jul 28 18:07:44.897439 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.897039 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:07:44.898845 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.898814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" event={"ID":"5505cefc-5b40-4812-be42-aa884e5e176c","Type":"ContainerStarted","Data":"b2f16bda7d103cf8c83fca1e7fa4be103b5b8f7d5d1621c847f2b426bc1fedb5"} Jul 28 18:07:44.898955 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.898872 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:07:44.900441 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.900419 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" event={"ID":"4ac59740-707e-4a48-8dec-ad3a21a89be0","Type":"ContainerStarted","Data":"007e1655cc2cc046b908f0e0c25ef366c65e668f2e38473f4b28104ba612ccd8"} Jul 28 18:07:44.902733 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.902712 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" event={"ID":"fc15845f-7f33-4334-8531-6da0357fd1b3","Type":"ContainerStarted","Data":"dd7b6828ae94bf7f8770f1452a140bb8db74f9d36fca2b27700bd469e8285560"} Jul 28 18:07:44.902843 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.902831 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:07:44.918469 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.918427 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" podStartSLOduration=1.196396937 podStartE2EDuration="4.918411958s" totalImagesPullingTime="3.722015021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:44.916781905 +0000 UTC m=+409.530066316" watchObservedRunningTime="2026-07-28 18:07:44.918411958 +0000 UTC m=+409.531696367" Jul 28 18:07:44.934564 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.934521 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79b97f775b-kg9ck" podStartSLOduration=2.933348124 podStartE2EDuration="6.934505696s" totalImagesPullingTime="4.001157572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:44.93267692 +0000 UTC m=+409.545961343" watchObservedRunningTime="2026-07-28 18:07:44.934505696 +0000 UTC m=+409.547790097" Jul 28 18:07:44.949483 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.949392 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" podStartSLOduration=3.333161496 podStartE2EDuration="7.949377376s" totalImagesPullingTime="4.61621588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:44.948448327 +0000 UTC m=+409.561732736" watchObservedRunningTime="2026-07-28 18:07:44.949377376 +0000 UTC m=+409.562661785" Jul 28 18:07:44.965982 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:44.965918 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" podStartSLOduration=2.301569514 podStartE2EDuration="7.965898282s" totalImagesPullingTime="5.664328768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:44.964728216 +0000 UTC m=+409.578012656" watchObservedRunningTime="2026-07-28 18:07:44.965898282 +0000 UTC m=+409.579182693" Jul 28 18:07:45.911109 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:07:45.911073 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-954rj" Jul 28 18:08:15.912000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:15.911964 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mnn5s" Jul 28 18:08:15.915623 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:15.915600 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwkbh" Jul 28 18:08:44.690356 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.690322 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7"] Jul 28 18:08:44.692899 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.692882 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.695572 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.695551 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 18:08:44.695665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.695630 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zlqmd\"" Jul 28 18:08:44.696728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.696706 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 18:08:44.702709 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.702689 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7"] Jul 28 18:08:44.793928 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.793889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.793928 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.793928 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrrrc\" (UniqueName: \"kubernetes.io/projected/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-kube-api-access-nrrrc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.794140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.794041 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.894769 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.894734 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.894769 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.894772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrrrc\" (UniqueName: \"kubernetes.io/projected/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-kube-api-access-nrrrc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.894972 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.894834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.895119 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.895099 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.895174 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.895154 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:44.904171 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:44.904140 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrrrc\" (UniqueName: \"kubernetes.io/projected/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-kube-api-access-nrrrc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:45.002784 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.002704 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:45.133672 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.133643 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7"] Jul 28 18:08:45.135656 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:08:45.135630 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9fea1b_bf7a_4e8b_9f27_5c23abea8f41.slice/crio-414d1ddf5d5660bd101ac307bc2fb8a166e5da8b4ba79246ddecf3efcc3079be WatchSource:0}: Error finding container 414d1ddf5d5660bd101ac307bc2fb8a166e5da8b4ba79246ddecf3efcc3079be: Status 404 returned error can't find the container with id 414d1ddf5d5660bd101ac307bc2fb8a166e5da8b4ba79246ddecf3efcc3079be Jul 28 18:08:45.199949 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.199919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" event={"ID":"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41","Type":"ContainerStarted","Data":"ed257c57948b72fd5d3d876ec8ca22e7e449e5f12c70d6afd8c55ef72e3ebe8a"} Jul 28 18:08:45.200054 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.199974 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" event={"ID":"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41","Type":"ContainerStarted","Data":"414d1ddf5d5660bd101ac307bc2fb8a166e5da8b4ba79246ddecf3efcc3079be"} Jul 28 18:08:45.496450 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.496418 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw"] Jul 28 18:08:45.499089 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.499068 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.508960 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.508935 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw"] Jul 28 18:08:45.601900 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.601870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.602067 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.601905 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.602067 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.601991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmnvg\" (UniqueName: \"kubernetes.io/projected/71d4f93c-8915-4762-89a8-761f19435806-kube-api-access-pmnvg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.703206 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.703174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmnvg\" (UniqueName: \"kubernetes.io/projected/71d4f93c-8915-4762-89a8-761f19435806-kube-api-access-pmnvg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.703595 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.703261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.703595 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.703286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.703678 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.703657 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.703678 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.703672 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.712167 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.712147 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmnvg\" (UniqueName: \"kubernetes.io/projected/71d4f93c-8915-4762-89a8-761f19435806-kube-api-access-pmnvg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.807927 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.807864 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:45.943854 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:45.943825 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw"] Jul 28 18:08:45.947557 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:08:45.947528 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d4f93c_8915_4762_89a8_761f19435806.slice/crio-bfae3e93dbfa4c9997ed863fcbadd590188c85b3c58138125419a0f2e622b2ce WatchSource:0}: Error finding container bfae3e93dbfa4c9997ed863fcbadd590188c85b3c58138125419a0f2e622b2ce: Status 404 returned error can't find the container with id bfae3e93dbfa4c9997ed863fcbadd590188c85b3c58138125419a0f2e622b2ce Jul 28 18:08:46.094107 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.094076 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v"] Jul 28 18:08:46.096623 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.096606 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.106533 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.106502 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v"] Jul 28 18:08:46.206204 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.206169 2569 generic.go:356] "Generic (PLEG): container finished" podID="71d4f93c-8915-4762-89a8-761f19435806" containerID="b5dd8290abc75a5849937b44b2eb02b4912c259adc7e1d6ff91a326c9e0fd18c" exitCode=0 Jul 28 18:08:46.206413 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.206260 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" event={"ID":"71d4f93c-8915-4762-89a8-761f19435806","Type":"ContainerDied","Data":"b5dd8290abc75a5849937b44b2eb02b4912c259adc7e1d6ff91a326c9e0fd18c"} Jul 28 18:08:46.206413 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.206308 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" event={"ID":"71d4f93c-8915-4762-89a8-761f19435806","Type":"ContainerStarted","Data":"bfae3e93dbfa4c9997ed863fcbadd590188c85b3c58138125419a0f2e622b2ce"} Jul 28 18:08:46.207545 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.207195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blrv\" (UniqueName: \"kubernetes.io/projected/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-kube-api-access-2blrv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.207545 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.207243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.207545 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.207361 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.207832 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.207809 2569 generic.go:356] "Generic (PLEG): container finished" podID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerID="ed257c57948b72fd5d3d876ec8ca22e7e449e5f12c70d6afd8c55ef72e3ebe8a" exitCode=0 Jul 28 18:08:46.207875 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.207843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" event={"ID":"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41","Type":"ContainerDied","Data":"ed257c57948b72fd5d3d876ec8ca22e7e449e5f12c70d6afd8c55ef72e3ebe8a"} Jul 28 18:08:46.308205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.308165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blrv\" (UniqueName: \"kubernetes.io/projected/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-kube-api-access-2blrv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.308205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.308206 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.308491 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.308338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.308697 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.308677 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.308768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.308679 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.316700 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.316680 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blrv\" (UniqueName: \"kubernetes.io/projected/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-kube-api-access-2blrv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.417128 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.417049 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:46.498888 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.498859 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t"] Jul 28 18:08:46.502296 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.502274 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.511163 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.510947 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t"] Jul 28 18:08:46.550691 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.550667 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v"] Jul 28 18:08:46.552628 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:08:46.552602 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc0dbbc_8350_4d82_b3a6_a02ea061b488.slice/crio-b000976bbe51327453160a2b6b4181b28f2c32c891c9ff4ff9b1d1f98fcc3822 WatchSource:0}: Error finding container b000976bbe51327453160a2b6b4181b28f2c32c891c9ff4ff9b1d1f98fcc3822: Status 404 returned error can't find the container with id b000976bbe51327453160a2b6b4181b28f2c32c891c9ff4ff9b1d1f98fcc3822 Jul 28 18:08:46.610414 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.610389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.610544 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.610428 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.610544 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.610519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtjd\" (UniqueName: \"kubernetes.io/projected/967cfe7b-b368-4594-94e2-47d888747e32-kube-api-access-smtjd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.712028 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.711938 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.712504 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.712064 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtjd\" (UniqueName: \"kubernetes.io/projected/967cfe7b-b368-4594-94e2-47d888747e32-kube-api-access-smtjd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.712504 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.712148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.712504 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.712426 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.712504 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.712495 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.721095 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.721064 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtjd\" (UniqueName: \"kubernetes.io/projected/967cfe7b-b368-4594-94e2-47d888747e32-kube-api-access-smtjd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:46.814991 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:46.814959 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:47.125527 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:47.125502 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t"] Jul 28 18:08:47.210000 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:08:47.209970 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967cfe7b_b368_4594_94e2_47d888747e32.slice/crio-2d2f6c02e8d709a06bc7b911a7e95b952283f3af9b027868035e47f37728f925 WatchSource:0}: Error finding container 2d2f6c02e8d709a06bc7b911a7e95b952283f3af9b027868035e47f37728f925: Status 404 returned error can't find the container with id 2d2f6c02e8d709a06bc7b911a7e95b952283f3af9b027868035e47f37728f925 Jul 28 18:08:47.214717 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:47.214694 2569 generic.go:356] "Generic (PLEG): container finished" podID="71d4f93c-8915-4762-89a8-761f19435806" containerID="0c8964a331aa762cddc02ec41c25149866563d3cad89a33e4543f4dc5233d0b7" exitCode=0 Jul 28 18:08:47.214871 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:47.214772 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" event={"ID":"71d4f93c-8915-4762-89a8-761f19435806","Type":"ContainerDied","Data":"0c8964a331aa762cddc02ec41c25149866563d3cad89a33e4543f4dc5233d0b7"} Jul 28 18:08:47.216878 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:47.216855 2569 generic.go:356] "Generic (PLEG): container finished" podID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerID="d6888dcfbaf07c33db50d2b88dd8ece0bcddf52eafef91ec2ecfb8d18316da78" exitCode=0 Jul 28 18:08:47.216971 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:47.216939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" event={"ID":"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41","Type":"ContainerDied","Data":"d6888dcfbaf07c33db50d2b88dd8ece0bcddf52eafef91ec2ecfb8d18316da78"} Jul 28 18:08:47.218525 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:47.218502 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerID="a0e58387633ad370076630e3ab3b74e09f93f640eb64dd827676ba5bd2b5a8c2" exitCode=0 Jul 28 18:08:47.218606 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:47.218555 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" event={"ID":"5cc0dbbc-8350-4d82-b3a6-a02ea061b488","Type":"ContainerDied","Data":"a0e58387633ad370076630e3ab3b74e09f93f640eb64dd827676ba5bd2b5a8c2"} Jul 28 18:08:47.218606 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:47.218585 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" event={"ID":"5cc0dbbc-8350-4d82-b3a6-a02ea061b488","Type":"ContainerStarted","Data":"b000976bbe51327453160a2b6b4181b28f2c32c891c9ff4ff9b1d1f98fcc3822"} Jul 28 18:08:48.226406 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.226321 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerID="3d13be0ec6c17c36b37405b6f5bad56a84c4569a02f19430e0bec986af6c0096" exitCode=0 Jul 28 18:08:48.226823 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.226400 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" event={"ID":"5cc0dbbc-8350-4d82-b3a6-a02ea061b488","Type":"ContainerDied","Data":"3d13be0ec6c17c36b37405b6f5bad56a84c4569a02f19430e0bec986af6c0096"} Jul 28 18:08:48.228532 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.228484 2569 generic.go:356] "Generic (PLEG): container finished" podID="71d4f93c-8915-4762-89a8-761f19435806" containerID="43c6da3450ea56e89962a78fae2dbe8565b4e2330b4c8a639173c40bf9b7ee79" exitCode=0 Jul 28 18:08:48.228660 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.228570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" event={"ID":"71d4f93c-8915-4762-89a8-761f19435806","Type":"ContainerDied","Data":"43c6da3450ea56e89962a78fae2dbe8565b4e2330b4c8a639173c40bf9b7ee79"} Jul 28 18:08:48.229869 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.229842 2569 generic.go:356] "Generic (PLEG): container finished" podID="967cfe7b-b368-4594-94e2-47d888747e32" containerID="9904691e6a3f50f3bf1925f1503e97b9640ee83a108f0052b9d04050e5179b0c" exitCode=0 Jul 28 18:08:48.230016 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.229954 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" event={"ID":"967cfe7b-b368-4594-94e2-47d888747e32","Type":"ContainerDied","Data":"9904691e6a3f50f3bf1925f1503e97b9640ee83a108f0052b9d04050e5179b0c"} Jul 28 18:08:48.230096 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.230061 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" event={"ID":"967cfe7b-b368-4594-94e2-47d888747e32","Type":"ContainerStarted","Data":"2d2f6c02e8d709a06bc7b911a7e95b952283f3af9b027868035e47f37728f925"} Jul 28 18:08:48.232209 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.232181 2569 generic.go:356] "Generic (PLEG): container finished" podID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerID="1cf2bbad7a4e1b96c34bd2466bcd4bd3a30dde63c9fdcf824bb665759161f676" exitCode=0 Jul 28 18:08:48.232295 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:48.232215 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" event={"ID":"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41","Type":"ContainerDied","Data":"1cf2bbad7a4e1b96c34bd2466bcd4bd3a30dde63c9fdcf824bb665759161f676"} Jul 28 18:08:49.239435 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.239403 2569 generic.go:356] "Generic (PLEG): container finished" podID="967cfe7b-b368-4594-94e2-47d888747e32" containerID="b289a06eab65cdf3ec0472225e1e5632827fffd1d79c2dc2d73d3132d4706496" exitCode=0 Jul 28 18:08:49.239817 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.239497 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" event={"ID":"967cfe7b-b368-4594-94e2-47d888747e32","Type":"ContainerDied","Data":"b289a06eab65cdf3ec0472225e1e5632827fffd1d79c2dc2d73d3132d4706496"} Jul 28 18:08:49.241718 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.241694 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerID="f0b78aaabc17342ef3efd03e6258e18a97a25ac4ff0cad92bd3aad69cb126451" exitCode=0 Jul 28 18:08:49.241827 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.241771 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" event={"ID":"5cc0dbbc-8350-4d82-b3a6-a02ea061b488","Type":"ContainerDied","Data":"f0b78aaabc17342ef3efd03e6258e18a97a25ac4ff0cad92bd3aad69cb126451"} Jul 28 18:08:49.389572 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.389546 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:49.403050 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.403025 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:49.538205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.538176 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-util\") pod \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " Jul 28 18:08:49.538370 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.538272 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmnvg\" (UniqueName: \"kubernetes.io/projected/71d4f93c-8915-4762-89a8-761f19435806-kube-api-access-pmnvg\") pod \"71d4f93c-8915-4762-89a8-761f19435806\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " Jul 28 18:08:49.538370 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.538302 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-bundle\") pod \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " Jul 28 18:08:49.538370 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.538331 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-util\") pod \"71d4f93c-8915-4762-89a8-761f19435806\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " Jul 28 18:08:49.538513 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.538402 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrrrc\" (UniqueName: \"kubernetes.io/projected/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-kube-api-access-nrrrc\") pod \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\" (UID: \"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41\") " Jul 28 18:08:49.538513 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.538418 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-bundle\") pod \"71d4f93c-8915-4762-89a8-761f19435806\" (UID: \"71d4f93c-8915-4762-89a8-761f19435806\") " Jul 28 18:08:49.538885 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.538852 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-bundle" (OuterVolumeSpecName: "bundle") pod "0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" (UID: "0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:08:49.539064 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.539040 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-bundle" (OuterVolumeSpecName: "bundle") pod "71d4f93c-8915-4762-89a8-761f19435806" (UID: "71d4f93c-8915-4762-89a8-761f19435806"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:08:49.540710 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.540681 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-kube-api-access-nrrrc" (OuterVolumeSpecName: "kube-api-access-nrrrc") pod "0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" (UID: "0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41"). InnerVolumeSpecName "kube-api-access-nrrrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:08:49.540843 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.540825 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d4f93c-8915-4762-89a8-761f19435806-kube-api-access-pmnvg" (OuterVolumeSpecName: "kube-api-access-pmnvg") pod "71d4f93c-8915-4762-89a8-761f19435806" (UID: "71d4f93c-8915-4762-89a8-761f19435806"). InnerVolumeSpecName "kube-api-access-pmnvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:08:49.544300 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.544282 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-util" (OuterVolumeSpecName: "util") pod "71d4f93c-8915-4762-89a8-761f19435806" (UID: "71d4f93c-8915-4762-89a8-761f19435806"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:08:49.544630 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.544612 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-util" (OuterVolumeSpecName: "util") pod "0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" (UID: "0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:08:49.640049 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.639978 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrrrc\" (UniqueName: \"kubernetes.io/projected/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-kube-api-access-nrrrc\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:49.640049 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.640004 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:49.640049 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.640015 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:49.640049 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.640023 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmnvg\" (UniqueName: \"kubernetes.io/projected/71d4f93c-8915-4762-89a8-761f19435806-kube-api-access-pmnvg\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:49.640049 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.640032 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:49.640049 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:49.640040 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d4f93c-8915-4762-89a8-761f19435806-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:50.248600 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.248528 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" Jul 28 18:08:50.248985 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.248527 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw" event={"ID":"71d4f93c-8915-4762-89a8-761f19435806","Type":"ContainerDied","Data":"bfae3e93dbfa4c9997ed863fcbadd590188c85b3c58138125419a0f2e622b2ce"} Jul 28 18:08:50.248985 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.248659 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfae3e93dbfa4c9997ed863fcbadd590188c85b3c58138125419a0f2e622b2ce" Jul 28 18:08:50.250831 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.250798 2569 generic.go:356] "Generic (PLEG): container finished" podID="967cfe7b-b368-4594-94e2-47d888747e32" containerID="0e9e694182aafee41b9a7a209769a3c9cde40f91285e29eb01790d6bbcd480da" exitCode=0 Jul 28 18:08:50.250960 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.250881 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" event={"ID":"967cfe7b-b368-4594-94e2-47d888747e32","Type":"ContainerDied","Data":"0e9e694182aafee41b9a7a209769a3c9cde40f91285e29eb01790d6bbcd480da"} Jul 28 18:08:50.252659 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.252641 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" Jul 28 18:08:50.252747 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.252676 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7" event={"ID":"0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41","Type":"ContainerDied","Data":"414d1ddf5d5660bd101ac307bc2fb8a166e5da8b4ba79246ddecf3efcc3079be"} Jul 28 18:08:50.252747 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.252709 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="414d1ddf5d5660bd101ac307bc2fb8a166e5da8b4ba79246ddecf3efcc3079be" Jul 28 18:08:50.384914 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.384892 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:50.546815 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.546744 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2blrv\" (UniqueName: \"kubernetes.io/projected/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-kube-api-access-2blrv\") pod \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " Jul 28 18:08:50.546815 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.546784 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-util\") pod \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " Jul 28 18:08:50.546990 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.546850 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-bundle\") pod \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\" (UID: \"5cc0dbbc-8350-4d82-b3a6-a02ea061b488\") " Jul 28 18:08:50.547372 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.547339 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-bundle" (OuterVolumeSpecName: "bundle") pod "5cc0dbbc-8350-4d82-b3a6-a02ea061b488" (UID: "5cc0dbbc-8350-4d82-b3a6-a02ea061b488"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:08:50.548848 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.548821 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-kube-api-access-2blrv" (OuterVolumeSpecName: "kube-api-access-2blrv") pod "5cc0dbbc-8350-4d82-b3a6-a02ea061b488" (UID: "5cc0dbbc-8350-4d82-b3a6-a02ea061b488"). InnerVolumeSpecName "kube-api-access-2blrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:08:50.552265 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.552243 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-util" (OuterVolumeSpecName: "util") pod "5cc0dbbc-8350-4d82-b3a6-a02ea061b488" (UID: "5cc0dbbc-8350-4d82-b3a6-a02ea061b488"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:08:50.647804 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.647772 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:50.647804 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.647797 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:50.647804 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:50.647806 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blrv\" (UniqueName: \"kubernetes.io/projected/5cc0dbbc-8350-4d82-b3a6-a02ea061b488-kube-api-access-2blrv\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:51.259423 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.259389 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" Jul 28 18:08:51.259423 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.259401 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v" event={"ID":"5cc0dbbc-8350-4d82-b3a6-a02ea061b488","Type":"ContainerDied","Data":"b000976bbe51327453160a2b6b4181b28f2c32c891c9ff4ff9b1d1f98fcc3822"} Jul 28 18:08:51.259900 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.259453 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b000976bbe51327453160a2b6b4181b28f2c32c891c9ff4ff9b1d1f98fcc3822" Jul 28 18:08:51.391799 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.391774 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:08:51.555817 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.555725 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-bundle\") pod \"967cfe7b-b368-4594-94e2-47d888747e32\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " Jul 28 18:08:51.555981 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.555849 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smtjd\" (UniqueName: \"kubernetes.io/projected/967cfe7b-b368-4594-94e2-47d888747e32-kube-api-access-smtjd\") pod \"967cfe7b-b368-4594-94e2-47d888747e32\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " Jul 28 18:08:51.555981 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.555890 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-util\") pod \"967cfe7b-b368-4594-94e2-47d888747e32\" (UID: \"967cfe7b-b368-4594-94e2-47d888747e32\") " Jul 28 18:08:51.556469 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.556433 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-bundle" (OuterVolumeSpecName: "bundle") pod "967cfe7b-b368-4594-94e2-47d888747e32" (UID: "967cfe7b-b368-4594-94e2-47d888747e32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:08:51.557923 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.557902 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967cfe7b-b368-4594-94e2-47d888747e32-kube-api-access-smtjd" (OuterVolumeSpecName: "kube-api-access-smtjd") pod "967cfe7b-b368-4594-94e2-47d888747e32" (UID: "967cfe7b-b368-4594-94e2-47d888747e32"). InnerVolumeSpecName "kube-api-access-smtjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:08:51.560942 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.560916 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-util" (OuterVolumeSpecName: "util") pod "967cfe7b-b368-4594-94e2-47d888747e32" (UID: "967cfe7b-b368-4594-94e2-47d888747e32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:08:51.656981 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.656926 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smtjd\" (UniqueName: \"kubernetes.io/projected/967cfe7b-b368-4594-94e2-47d888747e32-kube-api-access-smtjd\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:51.656981 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.656975 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-util\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:51.656981 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:51.656990 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/967cfe7b-b368-4594-94e2-47d888747e32-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:08:52.266373 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:52.266284 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" event={"ID":"967cfe7b-b368-4594-94e2-47d888747e32","Type":"ContainerDied","Data":"2d2f6c02e8d709a06bc7b911a7e95b952283f3af9b027868035e47f37728f925"} Jul 28 18:08:52.266373 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:52.266330 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d2f6c02e8d709a06bc7b911a7e95b952283f3af9b027868035e47f37728f925" Jul 28 18:08:52.266373 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:08:52.266329 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t" Jul 28 18:09:13.064790 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.064753 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9"] Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065130 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerName="pull" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065141 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerName="pull" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065155 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967cfe7b-b368-4594-94e2-47d888747e32" containerName="pull" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065160 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967cfe7b-b368-4594-94e2-47d888747e32" containerName="pull" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065171 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967cfe7b-b368-4594-94e2-47d888747e32" containerName="util" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065180 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967cfe7b-b368-4594-94e2-47d888747e32" containerName="util" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065193 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d4f93c-8915-4762-89a8-761f19435806" containerName="util" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065198 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d4f93c-8915-4762-89a8-761f19435806" containerName="util" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065203 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerName="extract" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065208 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerName="extract" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065217 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerName="pull" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065221 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerName="pull" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065248 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerName="util" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065256 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerName="util" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065266 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d4f93c-8915-4762-89a8-761f19435806" containerName="pull" Jul 28 18:09:13.065269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065273 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d4f93c-8915-4762-89a8-761f19435806" containerName="pull" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065279 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967cfe7b-b368-4594-94e2-47d888747e32" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065284 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967cfe7b-b368-4594-94e2-47d888747e32" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065295 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d4f93c-8915-4762-89a8-761f19435806" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065299 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d4f93c-8915-4762-89a8-761f19435806" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065305 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerName="util" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065312 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerName="util" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065316 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065321 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065372 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc0dbbc-8350-4d82-b3a6-a02ea061b488" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065380 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="967cfe7b-b368-4594-94e2-47d888747e32" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065387 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41" containerName="extract" Jul 28 18:09:13.065728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.065394 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d4f93c-8915-4762-89a8-761f19435806" containerName="extract" Jul 28 18:09:13.069637 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.069622 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:13.072794 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.072768 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nmb25\"" Jul 28 18:09:13.072923 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.072768 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 18:09:13.073855 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.073835 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 18:09:13.079854 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.079833 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9"] Jul 28 18:09:13.140109 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.140087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5rz\" (UniqueName: \"kubernetes.io/projected/b9c9e492-cad7-4046-9591-e7723a9af520-kube-api-access-jc5rz\") pod \"limitador-operator-controller-manager-859d7b78bd-s8fd9\" (UID: \"b9c9e492-cad7-4046-9591-e7723a9af520\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:13.241046 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.241016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5rz\" (UniqueName: \"kubernetes.io/projected/b9c9e492-cad7-4046-9591-e7723a9af520-kube-api-access-jc5rz\") pod \"limitador-operator-controller-manager-859d7b78bd-s8fd9\" (UID: \"b9c9e492-cad7-4046-9591-e7723a9af520\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:13.252411 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.252377 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5rz\" (UniqueName: \"kubernetes.io/projected/b9c9e492-cad7-4046-9591-e7723a9af520-kube-api-access-jc5rz\") pod \"limitador-operator-controller-manager-859d7b78bd-s8fd9\" (UID: \"b9c9e492-cad7-4046-9591-e7723a9af520\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:13.382365 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.382330 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:13.534291 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:13.534269 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9"] Jul 28 18:09:13.535831 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:09:13.535803 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c9e492_cad7_4046_9591_e7723a9af520.slice/crio-3ea7fa77f230ec8a2308fed53159a275c3b3a5375c864e39b46ddf1f2e318131 WatchSource:0}: Error finding container 3ea7fa77f230ec8a2308fed53159a275c3b3a5375c864e39b46ddf1f2e318131: Status 404 returned error can't find the container with id 3ea7fa77f230ec8a2308fed53159a275c3b3a5375c864e39b46ddf1f2e318131 Jul 28 18:09:14.385677 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:14.385644 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" event={"ID":"b9c9e492-cad7-4046-9591-e7723a9af520","Type":"ContainerStarted","Data":"3ea7fa77f230ec8a2308fed53159a275c3b3a5375c864e39b46ddf1f2e318131"} Jul 28 18:09:16.398088 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:16.398035 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" event={"ID":"b9c9e492-cad7-4046-9591-e7723a9af520","Type":"ContainerStarted","Data":"b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc"} Jul 28 18:09:16.398538 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:16.398109 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:16.421397 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:16.421352 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" podStartSLOduration=1.543585279 podStartE2EDuration="3.421337592s" totalImagesPullingTime="1.877752313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:16.420405747 +0000 UTC m=+501.033690159" watchObservedRunningTime="2026-07-28 18:09:16.421337592 +0000 UTC m=+501.034622001" Jul 28 18:09:27.405160 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:27.405131 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:32.456690 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.456646 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9"] Jul 28 18:09:32.457206 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.456864 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" containerName="manager" containerID="cri-o://b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc" gracePeriod=2 Jul 28 18:09:32.460028 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:09:32.459978 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8fd9\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:32.461811 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.461781 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9"] Jul 28 18:09:32.486396 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.486361 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx"] Jul 28 18:09:32.486929 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.486902 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" containerName="manager" Jul 28 18:09:32.486929 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.486931 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" containerName="manager" Jul 28 18:09:32.487063 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.487027 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" containerName="manager" Jul 28 18:09:32.490085 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.490067 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" Jul 28 18:09:32.492738 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:09:32.492706 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8fd9\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:32.505944 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.505919 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx"] Jul 28 18:09:32.612378 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.612344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfzck\" (UniqueName: \"kubernetes.io/projected/7cc8d485-ec21-41a2-a692-bd9e04394018-kube-api-access-wfzck\") pod \"limitador-operator-controller-manager-859d7b78bd-tlgkx\" (UID: \"7cc8d485-ec21-41a2-a692-bd9e04394018\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" Jul 28 18:09:32.693479 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.693453 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:32.695798 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:09:32.695774 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8fd9\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:32.713287 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.713202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfzck\" (UniqueName: \"kubernetes.io/projected/7cc8d485-ec21-41a2-a692-bd9e04394018-kube-api-access-wfzck\") pod \"limitador-operator-controller-manager-859d7b78bd-tlgkx\" (UID: \"7cc8d485-ec21-41a2-a692-bd9e04394018\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" Jul 28 18:09:32.725402 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.725380 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfzck\" (UniqueName: \"kubernetes.io/projected/7cc8d485-ec21-41a2-a692-bd9e04394018-kube-api-access-wfzck\") pod \"limitador-operator-controller-manager-859d7b78bd-tlgkx\" (UID: \"7cc8d485-ec21-41a2-a692-bd9e04394018\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" Jul 28 18:09:32.813853 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.813826 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc5rz\" (UniqueName: \"kubernetes.io/projected/b9c9e492-cad7-4046-9591-e7723a9af520-kube-api-access-jc5rz\") pod \"b9c9e492-cad7-4046-9591-e7723a9af520\" (UID: \"b9c9e492-cad7-4046-9591-e7723a9af520\") " Jul 28 18:09:32.815788 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.815765 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c9e492-cad7-4046-9591-e7723a9af520-kube-api-access-jc5rz" (OuterVolumeSpecName: "kube-api-access-jc5rz") pod "b9c9e492-cad7-4046-9591-e7723a9af520" (UID: "b9c9e492-cad7-4046-9591-e7723a9af520"). InnerVolumeSpecName "kube-api-access-jc5rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:09:32.821911 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.821888 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" Jul 28 18:09:32.914543 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.914515 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc5rz\" (UniqueName: \"kubernetes.io/projected/b9c9e492-cad7-4046-9591-e7723a9af520-kube-api-access-jc5rz\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:09:32.978093 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:32.978008 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx"] Jul 28 18:09:32.988653 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:09:32.988629 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc8d485_ec21_41a2_a692_bd9e04394018.slice/crio-55a607d15cff5e93f1285dba4c44aa422ddb2125309c69bbd96e9e346e71ec1b WatchSource:0}: Error finding container 55a607d15cff5e93f1285dba4c44aa422ddb2125309c69bbd96e9e346e71ec1b: Status 404 returned error can't find the container with id 55a607d15cff5e93f1285dba4c44aa422ddb2125309c69bbd96e9e346e71ec1b Jul 28 18:09:33.493491 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.493458 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" event={"ID":"7cc8d485-ec21-41a2-a692-bd9e04394018","Type":"ContainerStarted","Data":"e46e894fbc4a2d97a0ac8db670f93da51ace4ae85a3eeb635f4d498008965dcc"} Jul 28 18:09:33.493934 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.493510 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" event={"ID":"7cc8d485-ec21-41a2-a692-bd9e04394018","Type":"ContainerStarted","Data":"55a607d15cff5e93f1285dba4c44aa422ddb2125309c69bbd96e9e346e71ec1b"} Jul 28 18:09:33.493934 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.493562 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" Jul 28 18:09:33.494931 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.494911 2569 generic.go:356] "Generic (PLEG): container finished" podID="b9c9e492-cad7-4046-9591-e7723a9af520" containerID="b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc" exitCode=0 Jul 28 18:09:33.495032 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.494951 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:33.495032 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.495003 2569 scope.go:117] "RemoveContainer" containerID="b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc" Jul 28 18:09:33.496278 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:09:33.496030 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8fd9\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:33.503997 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.503984 2569 scope.go:117] "RemoveContainer" containerID="b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc" Jul 28 18:09:33.504259 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:09:33.504220 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc\": container with ID starting with b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc not found: ID does not exist" containerID="b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc" Jul 28 18:09:33.504355 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.504267 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc"} err="failed to get container status \"b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc\": rpc error: code = NotFound desc = could not find container \"b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc\": container with ID starting with b38473ef48cda640192343176594c733088de59c2877ec3d7853ce70218697fc not found: ID does not exist" Jul 28 18:09:33.517419 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.517377 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" podStartSLOduration=1.517364212 podStartE2EDuration="1.517364212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:33.516454073 +0000 UTC m=+518.129738489" watchObservedRunningTime="2026-07-28 18:09:33.517364212 +0000 UTC m=+518.130648621" Jul 28 18:09:33.518902 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:09:33.518875 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8fd9\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8fd9" Jul 28 18:09:33.975306 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:33.975274 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c9e492-cad7-4046-9591-e7723a9af520" path="/var/lib/kubelet/pods/b9c9e492-cad7-4046-9591-e7723a9af520/volumes" Jul 28 18:09:44.504379 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:09:44.504345 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlgkx" Jul 28 18:10:17.063902 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.063868 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dsg2g"] Jul 28 18:10:17.068749 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.068726 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:17.071488 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.071466 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zlqmd\"" Jul 28 18:10:17.071741 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.071717 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 18:10:17.077687 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.077666 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dsg2g"] Jul 28 18:10:17.097098 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.097070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-config-file\") pod \"limitador-limitador-64c94fd64f-dsg2g\" (UID: \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:17.097265 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.097116 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xhq\" (UniqueName: \"kubernetes.io/projected/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-kube-api-access-t7xhq\") pod \"limitador-limitador-64c94fd64f-dsg2g\" (UID: \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:17.150736 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.150707 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dsg2g"] Jul 28 18:10:17.198382 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.198352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-config-file\") pod \"limitador-limitador-64c94fd64f-dsg2g\" (UID: \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:17.198539 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.198396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xhq\" (UniqueName: \"kubernetes.io/projected/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-kube-api-access-t7xhq\") pod \"limitador-limitador-64c94fd64f-dsg2g\" (UID: \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:17.198993 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.198975 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-config-file\") pod \"limitador-limitador-64c94fd64f-dsg2g\" (UID: \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:17.208877 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.208846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xhq\" (UniqueName: \"kubernetes.io/projected/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-kube-api-access-t7xhq\") pod \"limitador-limitador-64c94fd64f-dsg2g\" (UID: \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:17.379314 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.379281 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:17.519367 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.519312 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dsg2g"] Jul 28 18:10:17.521637 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:10:17.521607 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9a6bd9_1306_4f30_bc16_f11bc20521c9.slice/crio-7f8c70d7d43eb6af975e71c8beb24707ef8f09b02e156a1788a8740469de6ad5 WatchSource:0}: Error finding container 7f8c70d7d43eb6af975e71c8beb24707ef8f09b02e156a1788a8740469de6ad5: Status 404 returned error can't find the container with id 7f8c70d7d43eb6af975e71c8beb24707ef8f09b02e156a1788a8740469de6ad5 Jul 28 18:10:17.726393 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:17.726316 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" event={"ID":"7d9a6bd9-1306-4f30-bc16-f11bc20521c9","Type":"ContainerStarted","Data":"7f8c70d7d43eb6af975e71c8beb24707ef8f09b02e156a1788a8740469de6ad5"} Jul 28 18:10:20.746290 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:20.746256 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" event={"ID":"7d9a6bd9-1306-4f30-bc16-f11bc20521c9","Type":"ContainerStarted","Data":"e364339f3d4032bfe045217ccd0ee6f39f8a8c2789b94fdb48a36b1fb5f57a3f"} Jul 28 18:10:20.746679 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:20.746401 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:20.766713 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:20.766667 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" podStartSLOduration=0.950086508 podStartE2EDuration="3.766654487s" totalImagesPullingTime="2.816567979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:20.765463234 +0000 UTC m=+565.378747663" watchObservedRunningTime="2026-07-28 18:10:20.766654487 +0000 UTC m=+565.379938898" Jul 28 18:10:31.752463 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:31.752430 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:31.987900 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:31.987838 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dsg2g"] Jul 28 18:10:31.988123 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:31.988033 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" podUID="7d9a6bd9-1306-4f30-bc16-f11bc20521c9" containerName="limitador" containerID="cri-o://e364339f3d4032bfe045217ccd0ee6f39f8a8c2789b94fdb48a36b1fb5f57a3f" gracePeriod=30 Jul 28 18:10:32.821411 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:32.821379 2569 generic.go:356] "Generic (PLEG): container finished" podID="7d9a6bd9-1306-4f30-bc16-f11bc20521c9" containerID="e364339f3d4032bfe045217ccd0ee6f39f8a8c2789b94fdb48a36b1fb5f57a3f" exitCode=0 Jul 28 18:10:32.821846 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:32.821429 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" event={"ID":"7d9a6bd9-1306-4f30-bc16-f11bc20521c9","Type":"ContainerDied","Data":"e364339f3d4032bfe045217ccd0ee6f39f8a8c2789b94fdb48a36b1fb5f57a3f"} Jul 28 18:10:32.931607 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:32.931583 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:33.038308 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.038276 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7xhq\" (UniqueName: \"kubernetes.io/projected/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-kube-api-access-t7xhq\") pod \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\" (UID: \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\") " Jul 28 18:10:33.038474 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.038373 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-config-file\") pod \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\" (UID: \"7d9a6bd9-1306-4f30-bc16-f11bc20521c9\") " Jul 28 18:10:33.038725 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.038698 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-config-file" (OuterVolumeSpecName: "config-file") pod "7d9a6bd9-1306-4f30-bc16-f11bc20521c9" (UID: "7d9a6bd9-1306-4f30-bc16-f11bc20521c9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:10:33.040288 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.040267 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-kube-api-access-t7xhq" (OuterVolumeSpecName: "kube-api-access-t7xhq") pod "7d9a6bd9-1306-4f30-bc16-f11bc20521c9" (UID: "7d9a6bd9-1306-4f30-bc16-f11bc20521c9"). InnerVolumeSpecName "kube-api-access-t7xhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:10:33.139388 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.139353 2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-config-file\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:10:33.139388 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.139384 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xhq\" (UniqueName: \"kubernetes.io/projected/7d9a6bd9-1306-4f30-bc16-f11bc20521c9-kube-api-access-t7xhq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:10:33.827568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.827532 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" Jul 28 18:10:33.827568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.827557 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dsg2g" event={"ID":"7d9a6bd9-1306-4f30-bc16-f11bc20521c9","Type":"ContainerDied","Data":"7f8c70d7d43eb6af975e71c8beb24707ef8f09b02e156a1788a8740469de6ad5"} Jul 28 18:10:33.828070 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.827620 2569 scope.go:117] "RemoveContainer" containerID="e364339f3d4032bfe045217ccd0ee6f39f8a8c2789b94fdb48a36b1fb5f57a3f" Jul 28 18:10:33.850352 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.850329 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dsg2g"] Jul 28 18:10:33.854304 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.854283 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dsg2g"] Jul 28 18:10:33.975559 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:33.975529 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9a6bd9-1306-4f30-bc16-f11bc20521c9" path="/var/lib/kubelet/pods/7d9a6bd9-1306-4f30-bc16-f11bc20521c9/volumes" Jul 28 18:10:34.881282 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.881249 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d"] Jul 28 18:10:34.881632 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.881608 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9a6bd9-1306-4f30-bc16-f11bc20521c9" containerName="limitador" Jul 28 18:10:34.881632 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.881619 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9a6bd9-1306-4f30-bc16-f11bc20521c9" containerName="limitador" Jul 28 18:10:34.881713 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.881698 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9a6bd9-1306-4f30-bc16-f11bc20521c9" containerName="limitador" Jul 28 18:10:34.886131 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.886112 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:34.889365 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.889348 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 28 18:10:34.891942 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.891926 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tntgx\"" Jul 28 18:10:34.895424 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.895391 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 18:10:34.895977 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.895955 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:10:34.897823 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.897802 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d"] Jul 28 18:10:34.957418 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.957327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9p8f\" (UniqueName: \"kubernetes.io/projected/46f63170-4224-4610-bcaa-fc9c9047c27d-kube-api-access-s9p8f\") pod \"postgres-69fb4594cf-wlh9d\" (UID: \"46f63170-4224-4610-bcaa-fc9c9047c27d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:34.957418 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:34.957372 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46f63170-4224-4610-bcaa-fc9c9047c27d-data\") pod \"postgres-69fb4594cf-wlh9d\" (UID: \"46f63170-4224-4610-bcaa-fc9c9047c27d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:35.058225 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:35.058189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9p8f\" (UniqueName: \"kubernetes.io/projected/46f63170-4224-4610-bcaa-fc9c9047c27d-kube-api-access-s9p8f\") pod \"postgres-69fb4594cf-wlh9d\" (UID: \"46f63170-4224-4610-bcaa-fc9c9047c27d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:35.058412 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:35.058263 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46f63170-4224-4610-bcaa-fc9c9047c27d-data\") pod \"postgres-69fb4594cf-wlh9d\" (UID: \"46f63170-4224-4610-bcaa-fc9c9047c27d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:35.058628 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:35.058611 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46f63170-4224-4610-bcaa-fc9c9047c27d-data\") pod \"postgres-69fb4594cf-wlh9d\" (UID: \"46f63170-4224-4610-bcaa-fc9c9047c27d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:35.066961 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:35.066939 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9p8f\" (UniqueName: \"kubernetes.io/projected/46f63170-4224-4610-bcaa-fc9c9047c27d-kube-api-access-s9p8f\") pod \"postgres-69fb4594cf-wlh9d\" (UID: \"46f63170-4224-4610-bcaa-fc9c9047c27d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:35.196816 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:35.196784 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:35.336888 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:35.336862 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d"] Jul 28 18:10:35.338355 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:10:35.338329 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f63170_4224_4610_bcaa_fc9c9047c27d.slice/crio-74464eb9da9b6fdce5bbf5db681d54eddcfa93fbd8588dd9700d945a5e592308 WatchSource:0}: Error finding container 74464eb9da9b6fdce5bbf5db681d54eddcfa93fbd8588dd9700d945a5e592308: Status 404 returned error can't find the container with id 74464eb9da9b6fdce5bbf5db681d54eddcfa93fbd8588dd9700d945a5e592308 Jul 28 18:10:35.839618 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:35.839585 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" event={"ID":"46f63170-4224-4610-bcaa-fc9c9047c27d","Type":"ContainerStarted","Data":"74464eb9da9b6fdce5bbf5db681d54eddcfa93fbd8588dd9700d945a5e592308"} Jul 28 18:10:41.877298 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:41.877254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" event={"ID":"46f63170-4224-4610-bcaa-fc9c9047c27d","Type":"ContainerStarted","Data":"85798c16d7f066d3a1c79d7f6806ed02e36faae14372a9755b1876a54a4f7055"} Jul 28 18:10:41.877692 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:41.877360 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:41.894247 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:41.894169 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" podStartSLOduration=2.256354092 podStartE2EDuration="7.894151613s" totalImagesPullingTime="5.637797521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:41.893791746 +0000 UTC m=+586.507076160" watchObservedRunningTime="2026-07-28 18:10:41.894151613 +0000 UTC m=+586.507436022" Jul 28 18:10:47.911823 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:47.911794 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wlh9d" Jul 28 18:10:55.373704 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.373671 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7588b7fb5d-xg8j4"] Jul 28 18:10:55.385951 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.385926 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:55.390071 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.390034 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 18:10:55.390247 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.390093 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 18:10:55.390247 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.390042 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-twsgn\"" Jul 28 18:10:55.394683 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.394271 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7588b7fb5d-xg8j4"] Jul 28 18:10:55.445507 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.445474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm89v\" (UniqueName: \"kubernetes.io/projected/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-kube-api-access-nm89v\") pod \"maas-controller-7588b7fb5d-xg8j4\" (UID: \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\") " pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:55.445672 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.445550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-webhook-cert\") pod \"maas-controller-7588b7fb5d-xg8j4\" (UID: \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\") " pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:55.545954 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.545919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-webhook-cert\") pod \"maas-controller-7588b7fb5d-xg8j4\" (UID: \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\") " pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:55.546139 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.546033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm89v\" (UniqueName: \"kubernetes.io/projected/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-kube-api-access-nm89v\") pod \"maas-controller-7588b7fb5d-xg8j4\" (UID: \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\") " pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:55.548445 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.548419 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-webhook-cert\") pod \"maas-controller-7588b7fb5d-xg8j4\" (UID: \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\") " pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:55.559972 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.559948 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm89v\" (UniqueName: \"kubernetes.io/projected/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-kube-api-access-nm89v\") pod \"maas-controller-7588b7fb5d-xg8j4\" (UID: \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\") " pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:55.698866 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.698785 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:55.839828 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.839801 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7588b7fb5d-xg8j4"] Jul 28 18:10:55.841899 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:10:55.841842 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8cd5dc_4bd0_4c4b_ae1a_abb85f21aec3.slice/crio-13d45b970bc02f7c1db1d733e0c94c207456869a186e3e1da64fae56b43e62ae WatchSource:0}: Error finding container 13d45b970bc02f7c1db1d733e0c94c207456869a186e3e1da64fae56b43e62ae: Status 404 returned error can't find the container with id 13d45b970bc02f7c1db1d733e0c94c207456869a186e3e1da64fae56b43e62ae Jul 28 18:10:55.845316 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.845296 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:10:55.849134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.849113 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:10:55.951412 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:55.951334 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" event={"ID":"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3","Type":"ContainerStarted","Data":"13d45b970bc02f7c1db1d733e0c94c207456869a186e3e1da64fae56b43e62ae"} Jul 28 18:10:58.556215 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:58.556190 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 18:10:58.971125 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:58.971095 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" event={"ID":"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3","Type":"ContainerStarted","Data":"0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62"} Jul 28 18:10:58.971309 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:58.971224 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:10:58.996718 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:10:58.996665 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" podStartSLOduration=1.290026836 podStartE2EDuration="3.996650155s" totalImagesPullingTime="2.706623319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:58.994480192 +0000 UTC m=+603.607764627" watchObservedRunningTime="2026-07-28 18:10:58.996650155 +0000 UTC m=+603.609934565" Jul 28 18:11:09.560841 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.560805 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7588b7fb5d-xg8j4"] Jul 28 18:11:09.563459 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.561014 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" podUID="3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3" containerName="manager" containerID="cri-o://0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62" gracePeriod=30 Jul 28 18:11:09.563459 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.563191 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:11:09.804307 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.804276 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:11:09.850987 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.850958 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm89v\" (UniqueName: \"kubernetes.io/projected/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-kube-api-access-nm89v\") pod \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\" (UID: \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\") " Jul 28 18:11:09.851190 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.851002 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-webhook-cert\") pod \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\" (UID: \"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3\") " Jul 28 18:11:09.853161 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.853133 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3" (UID: "3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:11:09.853268 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.853137 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-kube-api-access-nm89v" (OuterVolumeSpecName: "kube-api-access-nm89v") pod "3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3" (UID: "3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3"). InnerVolumeSpecName "kube-api-access-nm89v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:11:09.952472 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.952433 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-webhook-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:11:09.952472 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:09.952462 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm89v\" (UniqueName: \"kubernetes.io/projected/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3-kube-api-access-nm89v\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:11:10.033201 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.033168 2569 generic.go:356] "Generic (PLEG): container finished" podID="3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3" containerID="0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62" exitCode=0 Jul 28 18:11:10.033376 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.033213 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" Jul 28 18:11:10.033376 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.033223 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" event={"ID":"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3","Type":"ContainerDied","Data":"0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62"} Jul 28 18:11:10.033376 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.033305 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7588b7fb5d-xg8j4" event={"ID":"3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3","Type":"ContainerDied","Data":"13d45b970bc02f7c1db1d733e0c94c207456869a186e3e1da64fae56b43e62ae"} Jul 28 18:11:10.033376 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.033335 2569 scope.go:117] "RemoveContainer" containerID="0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62" Jul 28 18:11:10.041966 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.041951 2569 scope.go:117] "RemoveContainer" containerID="0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62" Jul 28 18:11:10.042203 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:11:10.042184 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62\": container with ID starting with 0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62 not found: ID does not exist" containerID="0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62" Jul 28 18:11:10.042301 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.042209 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62"} err="failed to get container status \"0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62\": rpc error: code = NotFound desc = could not find container \"0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62\": container with ID starting with 0b233c035054eb15e04929a95fb3ae4897ab38338fe43e8f79dfd8578b549c62 not found: ID does not exist" Jul 28 18:11:10.057191 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.054647 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7588b7fb5d-xg8j4"] Jul 28 18:11:10.060275 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:10.060254 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7588b7fb5d-xg8j4"] Jul 28 18:11:11.976140 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:11.976103 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3" path="/var/lib/kubelet/pods/3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3/volumes" Jul 28 18:11:34.633952 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.633863 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8"] Jul 28 18:11:34.634535 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.634400 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3" containerName="manager" Jul 28 18:11:34.634535 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.634419 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3" containerName="manager" Jul 28 18:11:34.634535 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.634522 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8cd5dc-4bd0-4c4b-ae1a-abb85f21aec3" containerName="manager" Jul 28 18:11:34.639203 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.639187 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.642122 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.642101 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gffgz\"" Jul 28 18:11:34.642268 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.642098 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 28 18:11:34.649274 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.649207 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8"] Jul 28 18:11:34.761876 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.761847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ad43d8-f609-4421-a3f7-15b27bc6cd51-maas-api-tls\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.762053 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.761903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zds\" (UniqueName: \"kubernetes.io/projected/43ad43d8-f609-4421-a3f7-15b27bc6cd51-kube-api-access-b8zds\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.762053 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.761975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ad43d8-f609-4421-a3f7-15b27bc6cd51-openshift-service-ca\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.863318 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.863282 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ad43d8-f609-4421-a3f7-15b27bc6cd51-maas-api-tls\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.863488 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.863345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zds\" (UniqueName: \"kubernetes.io/projected/43ad43d8-f609-4421-a3f7-15b27bc6cd51-kube-api-access-b8zds\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.863488 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.863382 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ad43d8-f609-4421-a3f7-15b27bc6cd51-openshift-service-ca\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.863965 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.863948 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ad43d8-f609-4421-a3f7-15b27bc6cd51-openshift-service-ca\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.865715 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.865691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ad43d8-f609-4421-a3f7-15b27bc6cd51-maas-api-tls\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.871391 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.871374 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zds\" (UniqueName: \"kubernetes.io/projected/43ad43d8-f609-4421-a3f7-15b27bc6cd51-kube-api-access-b8zds\") pod \"maas-api-6fd64c844b-dfmn8\" (UID: \"43ad43d8-f609-4421-a3f7-15b27bc6cd51\") " pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:34.951224 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:34.951130 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:35.090056 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.089995 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8"] Jul 28 18:11:35.091428 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:11:35.091400 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ad43d8_f609_4421_a3f7_15b27bc6cd51.slice/crio-0409fe1983278f938f8244b0bafa2e8a6550a13dd6dd6793d841428f5f704397 WatchSource:0}: Error finding container 0409fe1983278f938f8244b0bafa2e8a6550a13dd6dd6793d841428f5f704397: Status 404 returned error can't find the container with id 0409fe1983278f938f8244b0bafa2e8a6550a13dd6dd6793d841428f5f704397 Jul 28 18:11:35.093053 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.093035 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:11:35.169481 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.169449 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" event={"ID":"43ad43d8-f609-4421-a3f7-15b27bc6cd51","Type":"ContainerStarted","Data":"0409fe1983278f938f8244b0bafa2e8a6550a13dd6dd6793d841428f5f704397"} Jul 28 18:11:35.263768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.263690 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr"] Jul 28 18:11:35.266997 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.266977 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:35.269825 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.269802 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zqsl7\"" Jul 28 18:11:35.270180 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.270165 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 28 18:11:35.279676 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.279653 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr"] Jul 28 18:11:35.305124 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.305097 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9kkbx"] Jul 28 18:11:35.307563 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.307549 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:35.319037 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.318989 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9kkbx"] Jul 28 18:11:35.369592 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.369564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb1a8e1-f6ff-4ca0-8634-1492f212cad2-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-r2txr\" (UID: \"9eb1a8e1-f6ff-4ca0-8634-1492f212cad2\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:35.369746 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.369615 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqks\" (UniqueName: \"kubernetes.io/projected/9eb1a8e1-f6ff-4ca0-8634-1492f212cad2-kube-api-access-vzqks\") pod \"payload-pre-processing-687dbb6cf6-r2txr\" (UID: \"9eb1a8e1-f6ff-4ca0-8634-1492f212cad2\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:35.369746 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.369633 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c625k\" (UniqueName: \"kubernetes.io/projected/b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59-kube-api-access-c625k\") pod \"payload-processing-5cc6cd444b-9kkbx\" (UID: \"b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:35.369746 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.369714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9kkbx\" (UID: \"b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:35.471155 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.471126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb1a8e1-f6ff-4ca0-8634-1492f212cad2-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-r2txr\" (UID: \"9eb1a8e1-f6ff-4ca0-8634-1492f212cad2\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:35.471155 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.471179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqks\" (UniqueName: \"kubernetes.io/projected/9eb1a8e1-f6ff-4ca0-8634-1492f212cad2-kube-api-access-vzqks\") pod \"payload-pre-processing-687dbb6cf6-r2txr\" (UID: \"9eb1a8e1-f6ff-4ca0-8634-1492f212cad2\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:35.471155 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.471200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c625k\" (UniqueName: \"kubernetes.io/projected/b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59-kube-api-access-c625k\") pod \"payload-processing-5cc6cd444b-9kkbx\" (UID: \"b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:35.471562 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.471280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9kkbx\" (UID: \"b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:35.472000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.471980 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9kkbx\" (UID: \"b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:35.472072 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.472054 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb1a8e1-f6ff-4ca0-8634-1492f212cad2-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-r2txr\" (UID: \"9eb1a8e1-f6ff-4ca0-8634-1492f212cad2\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:35.481056 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.481035 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c625k\" (UniqueName: \"kubernetes.io/projected/b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59-kube-api-access-c625k\") pod \"payload-processing-5cc6cd444b-9kkbx\" (UID: \"b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:35.481663 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.481647 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqks\" (UniqueName: \"kubernetes.io/projected/9eb1a8e1-f6ff-4ca0-8634-1492f212cad2-kube-api-access-vzqks\") pod \"payload-pre-processing-687dbb6cf6-r2txr\" (UID: \"9eb1a8e1-f6ff-4ca0-8634-1492f212cad2\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:35.578649 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.578570 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:35.617750 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.617721 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:35.729950 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.729916 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr"] Jul 28 18:11:35.770724 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:35.770702 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9kkbx"] Jul 28 18:11:35.772127 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:11:35.772102 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f8aa4e_8c2b_478d_8c14_6d8d6518fc59.slice/crio-11cb785aebbf9b79feda5382d042905130503d3dfe3e6af86c804bd67526b4f9 WatchSource:0}: Error finding container 11cb785aebbf9b79feda5382d042905130503d3dfe3e6af86c804bd67526b4f9: Status 404 returned error can't find the container with id 11cb785aebbf9b79feda5382d042905130503d3dfe3e6af86c804bd67526b4f9 Jul 28 18:11:36.176291 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:36.176262 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" event={"ID":"b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59","Type":"ContainerStarted","Data":"11cb785aebbf9b79feda5382d042905130503d3dfe3e6af86c804bd67526b4f9"} Jul 28 18:11:36.177279 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:36.177256 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" event={"ID":"9eb1a8e1-f6ff-4ca0-8634-1492f212cad2","Type":"ContainerStarted","Data":"f5e2dd6d195d68c88b2a90f0cba8a4bbac356b4b433220b6bec0fdc8d4d40c11"} Jul 28 18:11:37.186595 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:37.186553 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" event={"ID":"43ad43d8-f609-4421-a3f7-15b27bc6cd51","Type":"ContainerStarted","Data":"b22ffcb50bae4701c96ecf809a605f1039116eab8d9590888028add273deaca9"} Jul 28 18:11:37.187081 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:37.187033 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:37.206150 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:37.206088 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" podStartSLOduration=1.257602364 podStartE2EDuration="3.206068648s" totalImagesPullingTime="1.948466284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:11:37.204811746 +0000 UTC m=+641.818096158" watchObservedRunningTime="2026-07-28 18:11:37.206068648 +0000 UTC m=+641.819353061" Jul 28 18:11:38.193162 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:38.193120 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" event={"ID":"b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59","Type":"ContainerStarted","Data":"f7f7d5be126a59fe9ea67a4cc272a5c826ae934f5bec129bae808a2373a80961"} Jul 28 18:11:38.193645 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:38.193258 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:11:38.194834 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:38.194814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" event={"ID":"9eb1a8e1-f6ff-4ca0-8634-1492f212cad2","Type":"ContainerStarted","Data":"69950ae4e3dc0eb2819d980bee6db3170205c70b66d3b02f806d5318383336d6"} Jul 28 18:11:38.195021 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:38.194998 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:38.212408 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:38.212356 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" podStartSLOduration=1.378434915 podStartE2EDuration="3.212339453s" totalImagesPullingTime="1.833904538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:11:38.211770825 +0000 UTC m=+642.825055235" watchObservedRunningTime="2026-07-28 18:11:38.212339453 +0000 UTC m=+642.825623867" Jul 28 18:11:38.227915 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:38.227869 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" podStartSLOduration=1.340209225 podStartE2EDuration="3.227856918s" totalImagesPullingTime="1.887647693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:11:38.22702731 +0000 UTC m=+642.840311716" watchObservedRunningTime="2026-07-28 18:11:38.227856918 +0000 UTC m=+642.841141328" Jul 28 18:11:43.200988 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:43.200960 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fd64c844b-dfmn8" Jul 28 18:11:49.201686 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:49.201654 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-r2txr" Jul 28 18:11:49.202069 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:11:49.201723 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9kkbx" Jul 28 18:12:24.186586 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.186548 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb"] Jul 28 18:12:24.190845 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.190811 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.195206 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.195171 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-87r8d\"" Jul 28 18:12:24.195206 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.195186 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 28 18:12:24.195426 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.195176 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 18:12:24.195426 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.195177 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 18:12:24.202293 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.202272 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb"] Jul 28 18:12:24.319677 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.319642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.319847 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.319686 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.319847 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.319775 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfws\" (UniqueName: \"kubernetes.io/projected/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-kube-api-access-fnfws\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.319847 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.319820 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.320004 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.319885 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.320004 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.319921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.420833 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.420797 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.421030 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.420844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.421030 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.420862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.421030 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.420891 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.421030 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.420930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.421030 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.420989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfws\" (UniqueName: \"kubernetes.io/projected/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-kube-api-access-fnfws\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.421335 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.421195 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.421392 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.421355 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.421433 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.421411 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.423179 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.423155 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.423472 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.423450 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.430414 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.430392 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfws\" (UniqueName: \"kubernetes.io/projected/8f6cd429-3f1b-480a-a40f-7cdedd3ebf02-kube-api-access-fnfws\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9hb\" (UID: \"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.502340 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.502254 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:24.641740 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:24.641715 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb"] Jul 28 18:12:24.643870 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:12:24.643843 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6cd429_3f1b_480a_a40f_7cdedd3ebf02.slice/crio-2c087af1a55985f411aa1333a3ebdec17bbf62ff0c42570823bc0b4218b3536a WatchSource:0}: Error finding container 2c087af1a55985f411aa1333a3ebdec17bbf62ff0c42570823bc0b4218b3536a: Status 404 returned error can't find the container with id 2c087af1a55985f411aa1333a3ebdec17bbf62ff0c42570823bc0b4218b3536a Jul 28 18:12:25.460167 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:25.460128 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" event={"ID":"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02","Type":"ContainerStarted","Data":"2c087af1a55985f411aa1333a3ebdec17bbf62ff0c42570823bc0b4218b3536a"} Jul 28 18:12:26.467037 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:26.466989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" event={"ID":"8f6cd429-3f1b-480a-a40f-7cdedd3ebf02","Type":"ContainerStarted","Data":"ac0d05f98ff5269f782312315ae603c41873de4aac4fd3ead662d3f3ad7fffc0"} Jul 28 18:12:26.487589 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:26.487537 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" podStartSLOduration=1.483121259 podStartE2EDuration="2.487522073s" totalImagesPullingTime="1.004400814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:12:26.486067362 +0000 UTC m=+691.099351772" watchObservedRunningTime="2026-07-28 18:12:26.487522073 +0000 UTC m=+691.100806483" Jul 28 18:12:34.502458 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:34.502417 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:34.502871 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:34.502482 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:34.515919 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:34.515894 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:34.527389 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:34.527369 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9hb" Jul 28 18:12:35.110984 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.110953 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq"] Jul 28 18:12:35.116615 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.116585 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.120148 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.120126 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 28 18:12:35.127535 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.127512 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq"] Jul 28 18:12:35.219840 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.219805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5bk9\" (UniqueName: \"kubernetes.io/projected/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-kube-api-access-m5bk9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.219995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.219853 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.219995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.219913 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.219995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.219946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.219995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.219961 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.219995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.219988 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.321384 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321348 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.321384 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.321608 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.321608 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321475 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5bk9\" (UniqueName: \"kubernetes.io/projected/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-kube-api-access-m5bk9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.321608 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321543 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.321608 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.321906 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321886 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.321906 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321896 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.322019 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.321953 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.323658 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.323636 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.323955 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.323937 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.330808 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.330790 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5bk9\" (UniqueName: \"kubernetes.io/projected/6cb10d46-8b88-4f53-82e4-4925dcd6b9a0-kube-api-access-m5bk9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lglmq\" (UID: \"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.429365 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.429284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:35.571006 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:35.570978 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq"] Jul 28 18:12:35.572549 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:12:35.572515 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb10d46_8b88_4f53_82e4_4925dcd6b9a0.slice/crio-376ff29aa0368891366ff13db70c172aaeff8407ba394337260cd76b4a0a4a4e WatchSource:0}: Error finding container 376ff29aa0368891366ff13db70c172aaeff8407ba394337260cd76b4a0a4a4e: Status 404 returned error can't find the container with id 376ff29aa0368891366ff13db70c172aaeff8407ba394337260cd76b4a0a4a4e Jul 28 18:12:36.531519 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:36.531480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" event={"ID":"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0","Type":"ContainerStarted","Data":"34e0121b7f454e3245ea102e344025872058d14283988a9fa18494a1a46a3b75"} Jul 28 18:12:36.531693 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:36.531539 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" event={"ID":"6cb10d46-8b88-4f53-82e4-4925dcd6b9a0","Type":"ContainerStarted","Data":"376ff29aa0368891366ff13db70c172aaeff8407ba394337260cd76b4a0a4a4e"} Jul 28 18:12:36.551451 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:36.551403 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" podStartSLOduration=1.368243645 podStartE2EDuration="1.551388451s" totalImagesPullingTime="183.144806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:12:36.550248891 +0000 UTC m=+701.163533295" watchObservedRunningTime="2026-07-28 18:12:36.551388451 +0000 UTC m=+701.164672861" Jul 28 18:12:45.429974 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:45.429939 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:45.430409 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:45.430014 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:45.443629 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:45.443606 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:12:45.593105 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:12:45.593076 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lglmq" Jul 28 18:15:55.891691 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:15:55.891607 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:15:55.892573 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:15:55.892551 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:20:55.929762 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:20:55.929733 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:20:55.931682 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:20:55.931664 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:25:55.977247 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:25:55.977211 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:25:55.978509 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:25:55.978489 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:29:29.156521 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.156436 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6"] Jul 28 18:29:29.159976 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.159960 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.162592 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.162568 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-314f4856-serving-cert\"" Jul 28 18:29:29.171174 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.171150 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6"] Jul 28 18:29:29.299734 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.299700 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a3e512-56f7-4b2b-bda6-53db70ded5e5-openshift-service-ca\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.299913 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.299778 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a3e512-56f7-4b2b-bda6-53db70ded5e5-maas-api-tls\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.299913 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.299823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hs7q\" (UniqueName: \"kubernetes.io/projected/83a3e512-56f7-4b2b-bda6-53db70ded5e5-kube-api-access-2hs7q\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.400870 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.400832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a3e512-56f7-4b2b-bda6-53db70ded5e5-openshift-service-ca\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.401047 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.400915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a3e512-56f7-4b2b-bda6-53db70ded5e5-maas-api-tls\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.401047 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.400961 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hs7q\" (UniqueName: \"kubernetes.io/projected/83a3e512-56f7-4b2b-bda6-53db70ded5e5-kube-api-access-2hs7q\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.401558 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.401520 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a3e512-56f7-4b2b-bda6-53db70ded5e5-openshift-service-ca\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.403309 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.403287 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a3e512-56f7-4b2b-bda6-53db70ded5e5-maas-api-tls\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.409281 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.409200 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hs7q\" (UniqueName: \"kubernetes.io/projected/83a3e512-56f7-4b2b-bda6-53db70ded5e5-kube-api-access-2hs7q\") pod \"maas-api-e2e-ait-314f4856-866b7db98d-hvvh6\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.472419 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.472390 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:29.617824 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.617674 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6"] Jul 28 18:29:29.620211 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:29:29.620180 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a3e512_56f7_4b2b_bda6_53db70ded5e5.slice/crio-1f2417e3d5d17c92d728ff72283ad134a72564ab4e08172ba2370f7ca4eefea1 WatchSource:0}: Error finding container 1f2417e3d5d17c92d728ff72283ad134a72564ab4e08172ba2370f7ca4eefea1: Status 404 returned error can't find the container with id 1f2417e3d5d17c92d728ff72283ad134a72564ab4e08172ba2370f7ca4eefea1 Jul 28 18:29:29.621692 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.621676 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:29:29.857300 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.857264 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4"] Jul 28 18:29:29.862008 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.861990 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:29.864622 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.864597 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-314f4856\"" Jul 28 18:29:29.864739 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.864682 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-314f4856-dockercfg-cgm8v\"" Jul 28 18:29:29.870800 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:29.870766 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4"] Jul 28 18:29:30.006339 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.006306 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59n9q\" (UniqueName: \"kubernetes.io/projected/1d122584-8bf6-4403-b954-a08ca461d19f-kube-api-access-59n9q\") pod \"payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4\" (UID: \"1d122584-8bf6-4403-b954-a08ca461d19f\") " pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:30.006508 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.006374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d122584-8bf6-4403-b954-a08ca461d19f-plugins-config-volume\") pod \"payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4\" (UID: \"1d122584-8bf6-4403-b954-a08ca461d19f\") " pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:30.107648 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.107566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d122584-8bf6-4403-b954-a08ca461d19f-plugins-config-volume\") pod \"payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4\" (UID: \"1d122584-8bf6-4403-b954-a08ca461d19f\") " pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:30.107816 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.107677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59n9q\" (UniqueName: \"kubernetes.io/projected/1d122584-8bf6-4403-b954-a08ca461d19f-kube-api-access-59n9q\") pod \"payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4\" (UID: \"1d122584-8bf6-4403-b954-a08ca461d19f\") " pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:30.108311 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.108292 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d122584-8bf6-4403-b954-a08ca461d19f-plugins-config-volume\") pod \"payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4\" (UID: \"1d122584-8bf6-4403-b954-a08ca461d19f\") " pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:30.115952 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.115934 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59n9q\" (UniqueName: \"kubernetes.io/projected/1d122584-8bf6-4403-b954-a08ca461d19f-kube-api-access-59n9q\") pod \"payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4\" (UID: \"1d122584-8bf6-4403-b954-a08ca461d19f\") " pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:30.123847 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.123817 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" event={"ID":"83a3e512-56f7-4b2b-bda6-53db70ded5e5","Type":"ContainerStarted","Data":"1f2417e3d5d17c92d728ff72283ad134a72564ab4e08172ba2370f7ca4eefea1"} Jul 28 18:29:30.174808 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.174786 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:30.316418 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:30.316393 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4"] Jul 28 18:29:30.319017 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:29:30.318995 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d122584_8bf6_4403_b954_a08ca461d19f.slice/crio-8e704db7d46f9bbf67a19f109156395d0aaa52e8a3b0bc478f31ba0c2e463c67 WatchSource:0}: Error finding container 8e704db7d46f9bbf67a19f109156395d0aaa52e8a3b0bc478f31ba0c2e463c67: Status 404 returned error can't find the container with id 8e704db7d46f9bbf67a19f109156395d0aaa52e8a3b0bc478f31ba0c2e463c67 Jul 28 18:29:31.130472 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:31.130435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" event={"ID":"83a3e512-56f7-4b2b-bda6-53db70ded5e5","Type":"ContainerStarted","Data":"89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48"} Jul 28 18:29:31.130655 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:31.130501 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:31.131896 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:31.131870 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" event={"ID":"1d122584-8bf6-4403-b954-a08ca461d19f","Type":"ContainerStarted","Data":"bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5"} Jul 28 18:29:31.132003 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:31.131916 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" event={"ID":"1d122584-8bf6-4403-b954-a08ca461d19f","Type":"ContainerStarted","Data":"8e704db7d46f9bbf67a19f109156395d0aaa52e8a3b0bc478f31ba0c2e463c67"} Jul 28 18:29:31.132003 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:31.131944 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:31.148301 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:31.148259 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" podStartSLOduration=1.6164093899999998 podStartE2EDuration="2.148222186s" totalImagesPullingTime="531.812796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:29:31.147254302 +0000 UTC m=+1715.760538705" watchObservedRunningTime="2026-07-28 18:29:31.148222186 +0000 UTC m=+1715.761506596" Jul 28 18:29:31.166135 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:31.166081 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" podStartSLOduration=2.166065739 podStartE2EDuration="2.166065739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:29:31.164316327 +0000 UTC m=+1715.777600738" watchObservedRunningTime="2026-07-28 18:29:31.166065739 +0000 UTC m=+1715.779350148" Jul 28 18:29:37.143995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:37.143963 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:42.140754 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:42.140726 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:59.583635 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.583598 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6"] Jul 28 18:29:59.584263 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.583806 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" podUID="83a3e512-56f7-4b2b-bda6-53db70ded5e5" containerName="maas-api" containerID="cri-o://89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48" gracePeriod=30 Jul 28 18:29:59.636603 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.636567 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4"] Jul 28 18:29:59.636871 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.636817 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" podUID="1d122584-8bf6-4403-b954-a08ca461d19f" containerName="payload-processing" containerID="cri-o://bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5" gracePeriod=30 Jul 28 18:29:59.902830 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.902786 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:29:59.931009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.930978 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:29:59.986842 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.986812 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a3e512-56f7-4b2b-bda6-53db70ded5e5-maas-api-tls\") pod \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " Jul 28 18:29:59.986842 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.986843 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hs7q\" (UniqueName: \"kubernetes.io/projected/83a3e512-56f7-4b2b-bda6-53db70ded5e5-kube-api-access-2hs7q\") pod \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " Jul 28 18:29:59.987038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.986876 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d122584-8bf6-4403-b954-a08ca461d19f-plugins-config-volume\") pod \"1d122584-8bf6-4403-b954-a08ca461d19f\" (UID: \"1d122584-8bf6-4403-b954-a08ca461d19f\") " Jul 28 18:29:59.987038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.986973 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a3e512-56f7-4b2b-bda6-53db70ded5e5-openshift-service-ca\") pod \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\" (UID: \"83a3e512-56f7-4b2b-bda6-53db70ded5e5\") " Jul 28 18:29:59.987038 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.987010 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59n9q\" (UniqueName: \"kubernetes.io/projected/1d122584-8bf6-4403-b954-a08ca461d19f-kube-api-access-59n9q\") pod \"1d122584-8bf6-4403-b954-a08ca461d19f\" (UID: \"1d122584-8bf6-4403-b954-a08ca461d19f\") " Jul 28 18:29:59.987368 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.987340 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d122584-8bf6-4403-b954-a08ca461d19f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d122584-8bf6-4403-b954-a08ca461d19f" (UID: "1d122584-8bf6-4403-b954-a08ca461d19f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:29:59.987368 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.987360 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a3e512-56f7-4b2b-bda6-53db70ded5e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83a3e512-56f7-4b2b-bda6-53db70ded5e5" (UID: "83a3e512-56f7-4b2b-bda6-53db70ded5e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:29:59.989114 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.989094 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d122584-8bf6-4403-b954-a08ca461d19f-kube-api-access-59n9q" (OuterVolumeSpecName: "kube-api-access-59n9q") pod "1d122584-8bf6-4403-b954-a08ca461d19f" (UID: "1d122584-8bf6-4403-b954-a08ca461d19f"). InnerVolumeSpecName "kube-api-access-59n9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:29:59.989568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.989549 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a3e512-56f7-4b2b-bda6-53db70ded5e5-kube-api-access-2hs7q" (OuterVolumeSpecName: "kube-api-access-2hs7q") pod "83a3e512-56f7-4b2b-bda6-53db70ded5e5" (UID: "83a3e512-56f7-4b2b-bda6-53db70ded5e5"). InnerVolumeSpecName "kube-api-access-2hs7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:29:59.989645 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:29:59.989550 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a3e512-56f7-4b2b-bda6-53db70ded5e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83a3e512-56f7-4b2b-bda6-53db70ded5e5" (UID: "83a3e512-56f7-4b2b-bda6-53db70ded5e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:30:00.087874 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.087838 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a3e512-56f7-4b2b-bda6-53db70ded5e5-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:30:00.087874 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.087871 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hs7q\" (UniqueName: \"kubernetes.io/projected/83a3e512-56f7-4b2b-bda6-53db70ded5e5-kube-api-access-2hs7q\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:30:00.087874 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.087884 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d122584-8bf6-4403-b954-a08ca461d19f-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:30:00.088107 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.087894 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a3e512-56f7-4b2b-bda6-53db70ded5e5-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:30:00.088107 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.087905 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59n9q\" (UniqueName: \"kubernetes.io/projected/1d122584-8bf6-4403-b954-a08ca461d19f-kube-api-access-59n9q\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:30:00.295571 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.295479 2569 generic.go:356] "Generic (PLEG): container finished" podID="1d122584-8bf6-4403-b954-a08ca461d19f" containerID="bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5" exitCode=0 Jul 28 18:30:00.295571 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.295527 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" Jul 28 18:30:00.295768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.295569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" event={"ID":"1d122584-8bf6-4403-b954-a08ca461d19f","Type":"ContainerDied","Data":"bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5"} Jul 28 18:30:00.295768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.295624 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4" event={"ID":"1d122584-8bf6-4403-b954-a08ca461d19f","Type":"ContainerDied","Data":"8e704db7d46f9bbf67a19f109156395d0aaa52e8a3b0bc478f31ba0c2e463c67"} Jul 28 18:30:00.295768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.295651 2569 scope.go:117] "RemoveContainer" containerID="bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5" Jul 28 18:30:00.296963 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.296940 2569 generic.go:356] "Generic (PLEG): container finished" podID="83a3e512-56f7-4b2b-bda6-53db70ded5e5" containerID="89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48" exitCode=0 Jul 28 18:30:00.297077 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.296973 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" event={"ID":"83a3e512-56f7-4b2b-bda6-53db70ded5e5","Type":"ContainerDied","Data":"89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48"} Jul 28 18:30:00.297077 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.297028 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" event={"ID":"83a3e512-56f7-4b2b-bda6-53db70ded5e5","Type":"ContainerDied","Data":"1f2417e3d5d17c92d728ff72283ad134a72564ab4e08172ba2370f7ca4eefea1"} Jul 28 18:30:00.297077 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.296987 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6" Jul 28 18:30:00.305273 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.305255 2569 scope.go:117] "RemoveContainer" containerID="bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5" Jul 28 18:30:00.305677 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:30:00.305651 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5\": container with ID starting with bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5 not found: ID does not exist" containerID="bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5" Jul 28 18:30:00.305764 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.305683 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5"} err="failed to get container status \"bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5\": rpc error: code = NotFound desc = could not find container \"bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5\": container with ID starting with bf76bbafd29f16b01337139c0489f4529fd1395486fa3969a21b71ffc7ee5bc5 not found: ID does not exist" Jul 28 18:30:00.305764 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.305698 2569 scope.go:117] "RemoveContainer" containerID="89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48" Jul 28 18:30:00.316356 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.316277 2569 scope.go:117] "RemoveContainer" containerID="89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48" Jul 28 18:30:00.316896 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:30:00.316871 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48\": container with ID starting with 89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48 not found: ID does not exist" containerID="89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48" Jul 28 18:30:00.316963 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.316906 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48"} err="failed to get container status \"89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48\": rpc error: code = NotFound desc = could not find container \"89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48\": container with ID starting with 89c3d17c4694c0f5b44e52358cc33917c5421f2954ca3fbd95b256feb3a2ac48 not found: ID does not exist" Jul 28 18:30:00.322370 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.322350 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6"] Jul 28 18:30:00.326397 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.326378 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-314f4856-866b7db98d-hvvh6"] Jul 28 18:30:00.338151 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.338124 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4"] Jul 28 18:30:00.341905 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:00.341884 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-314f4856-5d7c7c677d-lppq4"] Jul 28 18:30:01.976616 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:01.976584 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d122584-8bf6-4403-b954-a08ca461d19f" path="/var/lib/kubelet/pods/1d122584-8bf6-4403-b954-a08ca461d19f/volumes" Jul 28 18:30:01.976991 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:01.976970 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a3e512-56f7-4b2b-bda6-53db70ded5e5" path="/var/lib/kubelet/pods/83a3e512-56f7-4b2b-bda6-53db70ded5e5/volumes" Jul 28 18:30:28.185728 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.185691 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z"] Jul 28 18:30:28.186186 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.186101 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a3e512-56f7-4b2b-bda6-53db70ded5e5" containerName="maas-api" Jul 28 18:30:28.186186 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.186113 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a3e512-56f7-4b2b-bda6-53db70ded5e5" containerName="maas-api" Jul 28 18:30:28.186186 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.186128 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d122584-8bf6-4403-b954-a08ca461d19f" containerName="payload-processing" Jul 28 18:30:28.186186 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.186134 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d122584-8bf6-4403-b954-a08ca461d19f" containerName="payload-processing" Jul 28 18:30:28.186360 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.186214 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a3e512-56f7-4b2b-bda6-53db70ded5e5" containerName="maas-api" Jul 28 18:30:28.186360 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.186224 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d122584-8bf6-4403-b954-a08ca461d19f" containerName="payload-processing" Jul 28 18:30:28.189295 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.189280 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" Jul 28 18:30:28.191838 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.191815 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:30:28.198295 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.198273 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z"] Jul 28 18:30:28.227939 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.227909 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9839d2-36b1-4087-8518-8c243172651e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z\" (UID: \"4c9839d2-36b1-4087-8518-8c243172651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" Jul 28 18:30:28.329188 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.329140 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9839d2-36b1-4087-8518-8c243172651e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z\" (UID: \"4c9839d2-36b1-4087-8518-8c243172651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" Jul 28 18:30:28.329757 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.329736 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9839d2-36b1-4087-8518-8c243172651e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z\" (UID: \"4c9839d2-36b1-4087-8518-8c243172651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" Jul 28 18:30:28.501830 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.501757 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" Jul 28 18:30:28.637491 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:28.637453 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z"] Jul 28 18:30:28.638713 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:30:28.638687 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9839d2_36b1_4087_8518_8c243172651e.slice/crio-254848b5543d8cb3f664c18c4ce97b0934fa29ad232a026e30e75891dd33fc2d WatchSource:0}: Error finding container 254848b5543d8cb3f664c18c4ce97b0934fa29ad232a026e30e75891dd33fc2d: Status 404 returned error can't find the container with id 254848b5543d8cb3f664c18c4ce97b0934fa29ad232a026e30e75891dd33fc2d Jul 28 18:30:29.472661 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:29.472621 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" event={"ID":"4c9839d2-36b1-4087-8518-8c243172651e","Type":"ContainerStarted","Data":"254848b5543d8cb3f664c18c4ce97b0934fa29ad232a026e30e75891dd33fc2d"} Jul 28 18:30:31.486417 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:31.486379 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" event={"ID":"4c9839d2-36b1-4087-8518-8c243172651e","Type":"ContainerStarted","Data":"49c243e334a9c993d8d492b9f373cff7cdb0eb68e8dc203bc646bfb3fc326800"} Jul 28 18:30:31.503014 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:31.502927 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" podStartSLOduration=1.16433507 podStartE2EDuration="3.502911829s" totalImagesPullingTime="2.338576759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:31.501175306 +0000 UTC m=+1776.114459736" watchObservedRunningTime="2026-07-28 18:30:31.502911829 +0000 UTC m=+1776.116196280" Jul 28 18:30:32.493603 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:32.493566 2569 generic.go:356] "Generic (PLEG): container finished" podID="4c9839d2-36b1-4087-8518-8c243172651e" containerID="49c243e334a9c993d8d492b9f373cff7cdb0eb68e8dc203bc646bfb3fc326800" exitCode=0 Jul 28 18:30:32.493992 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:32.493598 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" event={"ID":"4c9839d2-36b1-4087-8518-8c243172651e","Type":"ContainerDied","Data":"49c243e334a9c993d8d492b9f373cff7cdb0eb68e8dc203bc646bfb3fc326800"} Jul 28 18:30:33.640469 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:33.640443 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" Jul 28 18:30:33.676398 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:33.676366 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9839d2-36b1-4087-8518-8c243172651e-maas-api-service-ca\") pod \"4c9839d2-36b1-4087-8518-8c243172651e\" (UID: \"4c9839d2-36b1-4087-8518-8c243172651e\") " Jul 28 18:30:33.676790 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:33.676761 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9839d2-36b1-4087-8518-8c243172651e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c9839d2-36b1-4087-8518-8c243172651e" (UID: "4c9839d2-36b1-4087-8518-8c243172651e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:30:33.778039 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:33.777954 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9839d2-36b1-4087-8518-8c243172651e-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:30:34.506374 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:34.506341 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" event={"ID":"4c9839d2-36b1-4087-8518-8c243172651e","Type":"ContainerDied","Data":"254848b5543d8cb3f664c18c4ce97b0934fa29ad232a026e30e75891dd33fc2d"} Jul 28 18:30:34.506558 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:34.506390 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z" Jul 28 18:30:34.506558 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:34.506393 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="254848b5543d8cb3f664c18c4ce97b0934fa29ad232a026e30e75891dd33fc2d" Jul 28 18:30:44.881616 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:44.881574 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z"] Jul 28 18:30:44.883924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:44.883900 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ba40d93a-rsv4z"] Jul 28 18:30:45.978405 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:45.978375 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c9839d2-36b1-4087-8518-8c243172651e" path="/var/lib/kubelet/pods/4c9839d2-36b1-4087-8518-8c243172651e/volumes" Jul 28 18:30:56.014975 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:56.014948 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:30:56.018611 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:30:56.018587 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:31:14.160440 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.160405 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8"] Jul 28 18:31:14.160854 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.160812 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c9839d2-36b1-4087-8518-8c243172651e" containerName="revoke-keys" Jul 28 18:31:14.160854 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.160823 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c9839d2-36b1-4087-8518-8c243172651e" containerName="revoke-keys" Jul 28 18:31:14.160929 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.160900 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c9839d2-36b1-4087-8518-8c243172651e" containerName="revoke-keys" Jul 28 18:31:14.168990 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.168963 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" Jul 28 18:31:14.171565 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.171537 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8"] Jul 28 18:31:14.171767 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.171620 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:31:14.246273 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.246222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f0a6a-dfbf-4565-bba1-9f33f286a39e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8\" (UID: \"659f0a6a-dfbf-4565-bba1-9f33f286a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" Jul 28 18:31:14.347443 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.347408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f0a6a-dfbf-4565-bba1-9f33f286a39e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8\" (UID: \"659f0a6a-dfbf-4565-bba1-9f33f286a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" Jul 28 18:31:14.348058 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.348037 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f0a6a-dfbf-4565-bba1-9f33f286a39e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8\" (UID: \"659f0a6a-dfbf-4565-bba1-9f33f286a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" Jul 28 18:31:14.480418 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.480332 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" Jul 28 18:31:14.615564 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.615532 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8"] Jul 28 18:31:14.616870 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:31:14.616837 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659f0a6a_dfbf_4565_bba1_9f33f286a39e.slice/crio-36fcac377df4b262c173326bee4a4905e261744c856753d5d0f53e1e87777b90 WatchSource:0}: Error finding container 36fcac377df4b262c173326bee4a4905e261744c856753d5d0f53e1e87777b90: Status 404 returned error can't find the container with id 36fcac377df4b262c173326bee4a4905e261744c856753d5d0f53e1e87777b90 Jul 28 18:31:14.731029 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:14.730945 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" event={"ID":"659f0a6a-dfbf-4565-bba1-9f33f286a39e","Type":"ContainerStarted","Data":"36fcac377df4b262c173326bee4a4905e261744c856753d5d0f53e1e87777b90"} Jul 28 18:31:15.738009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:15.737910 2569 generic.go:356] "Generic (PLEG): container finished" podID="659f0a6a-dfbf-4565-bba1-9f33f286a39e" containerID="bf0b91bc8abc9f779884ef2c2e65d89992ced135afee7d5f476d52fd787d49fb" exitCode=0 Jul 28 18:31:15.738009 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:15.737944 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" event={"ID":"659f0a6a-dfbf-4565-bba1-9f33f286a39e","Type":"ContainerDied","Data":"bf0b91bc8abc9f779884ef2c2e65d89992ced135afee7d5f476d52fd787d49fb"} Jul 28 18:31:16.872432 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:16.872405 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" Jul 28 18:31:16.967120 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:16.967091 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f0a6a-dfbf-4565-bba1-9f33f286a39e-maas-api-service-ca\") pod \"659f0a6a-dfbf-4565-bba1-9f33f286a39e\" (UID: \"659f0a6a-dfbf-4565-bba1-9f33f286a39e\") " Jul 28 18:31:16.967448 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:16.967426 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659f0a6a-dfbf-4565-bba1-9f33f286a39e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "659f0a6a-dfbf-4565-bba1-9f33f286a39e" (UID: "659f0a6a-dfbf-4565-bba1-9f33f286a39e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:31:17.068098 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:17.068012 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659f0a6a-dfbf-4565-bba1-9f33f286a39e-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:31:17.749532 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:17.749499 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" event={"ID":"659f0a6a-dfbf-4565-bba1-9f33f286a39e","Type":"ContainerDied","Data":"36fcac377df4b262c173326bee4a4905e261744c856753d5d0f53e1e87777b90"} Jul 28 18:31:17.749532 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:17.749523 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8" Jul 28 18:31:17.749746 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:17.749557 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36fcac377df4b262c173326bee4a4905e261744c856753d5d0f53e1e87777b90" Jul 28 18:31:19.232114 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:19.232085 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8"] Jul 28 18:31:19.234303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:19.234281 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b13f7e6e-qkcj8"] Jul 28 18:31:19.975993 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:19.975958 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="659f0a6a-dfbf-4565-bba1-9f33f286a39e" path="/var/lib/kubelet/pods/659f0a6a-dfbf-4565-bba1-9f33f286a39e/volumes" Jul 28 18:31:38.196246 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.196206 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw"] Jul 28 18:31:38.196677 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.196627 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659f0a6a-dfbf-4565-bba1-9f33f286a39e" containerName="revoke-keys" Jul 28 18:31:38.196677 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.196639 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="659f0a6a-dfbf-4565-bba1-9f33f286a39e" containerName="revoke-keys" Jul 28 18:31:38.196751 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.196716 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="659f0a6a-dfbf-4565-bba1-9f33f286a39e" containerName="revoke-keys" Jul 28 18:31:38.199784 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.199764 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" Jul 28 18:31:38.202113 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.202096 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:31:38.206854 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.206826 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw"] Jul 28 18:31:38.255513 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.255481 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2659a578-0952-41a9-80f3-98de97734302-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw\" (UID: \"2659a578-0952-41a9-80f3-98de97734302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" Jul 28 18:31:38.356242 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.356205 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2659a578-0952-41a9-80f3-98de97734302-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw\" (UID: \"2659a578-0952-41a9-80f3-98de97734302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" Jul 28 18:31:38.356778 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.356758 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2659a578-0952-41a9-80f3-98de97734302-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw\" (UID: \"2659a578-0952-41a9-80f3-98de97734302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" Jul 28 18:31:38.511373 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.511295 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" Jul 28 18:31:38.647738 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.647707 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw"] Jul 28 18:31:38.648109 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:31:38.648084 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2659a578_0952_41a9_80f3_98de97734302.slice/crio-7692a22c1372e77b585160fcbcac4f5c2fc80cd8baf6141e6046cf832473bf13 WatchSource:0}: Error finding container 7692a22c1372e77b585160fcbcac4f5c2fc80cd8baf6141e6046cf832473bf13: Status 404 returned error can't find the container with id 7692a22c1372e77b585160fcbcac4f5c2fc80cd8baf6141e6046cf832473bf13 Jul 28 18:31:38.874103 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:38.874075 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" event={"ID":"2659a578-0952-41a9-80f3-98de97734302","Type":"ContainerStarted","Data":"7692a22c1372e77b585160fcbcac4f5c2fc80cd8baf6141e6046cf832473bf13"} Jul 28 18:31:39.880913 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:39.880879 2569 generic.go:356] "Generic (PLEG): container finished" podID="2659a578-0952-41a9-80f3-98de97734302" containerID="b16cbdccb39e8e384328bdfbd97250acfe14a730e8b64a0e7881754d3ccb9e92" exitCode=0 Jul 28 18:31:39.881376 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:39.880929 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" event={"ID":"2659a578-0952-41a9-80f3-98de97734302","Type":"ContainerDied","Data":"b16cbdccb39e8e384328bdfbd97250acfe14a730e8b64a0e7881754d3ccb9e92"} Jul 28 18:31:41.027953 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:41.027930 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" Jul 28 18:31:41.081634 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:41.081602 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2659a578-0952-41a9-80f3-98de97734302-maas-api-service-ca\") pod \"2659a578-0952-41a9-80f3-98de97734302\" (UID: \"2659a578-0952-41a9-80f3-98de97734302\") " Jul 28 18:31:41.082303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:41.081959 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2659a578-0952-41a9-80f3-98de97734302-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2659a578-0952-41a9-80f3-98de97734302" (UID: "2659a578-0952-41a9-80f3-98de97734302"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:31:41.082303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:41.082078 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2659a578-0952-41a9-80f3-98de97734302-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:31:41.894626 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:41.894593 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" Jul 28 18:31:41.894813 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:41.894635 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw" event={"ID":"2659a578-0952-41a9-80f3-98de97734302","Type":"ContainerDied","Data":"7692a22c1372e77b585160fcbcac4f5c2fc80cd8baf6141e6046cf832473bf13"} Jul 28 18:31:41.894813 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:41.894713 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7692a22c1372e77b585160fcbcac4f5c2fc80cd8baf6141e6046cf832473bf13" Jul 28 18:31:43.609460 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:43.609426 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw"] Jul 28 18:31:43.611656 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:43.611634 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12084f6d-ff4sw"] Jul 28 18:31:43.976634 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:31:43.976541 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2659a578-0952-41a9-80f3-98de97734302" path="/var/lib/kubelet/pods/2659a578-0952-41a9-80f3-98de97734302/volumes" Jul 28 18:34:02.001677 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.001597 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz"] Jul 28 18:34:02.002129 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.002002 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2659a578-0952-41a9-80f3-98de97734302" containerName="revoke-keys" Jul 28 18:34:02.002129 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.002013 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2659a578-0952-41a9-80f3-98de97734302" containerName="revoke-keys" Jul 28 18:34:02.002129 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.002076 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2659a578-0952-41a9-80f3-98de97734302" containerName="revoke-keys" Jul 28 18:34:02.005179 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.005164 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" Jul 28 18:34:02.007949 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.007927 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:34:02.013275 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.013254 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz"] Jul 28 18:34:02.074591 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.074552 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a101bf-a426-401a-9a56-08b34eb1cc5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz\" (UID: \"03a101bf-a426-401a-9a56-08b34eb1cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" Jul 28 18:34:02.175146 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.175111 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a101bf-a426-401a-9a56-08b34eb1cc5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz\" (UID: \"03a101bf-a426-401a-9a56-08b34eb1cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" Jul 28 18:34:02.175740 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.175719 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a101bf-a426-401a-9a56-08b34eb1cc5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz\" (UID: \"03a101bf-a426-401a-9a56-08b34eb1cc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" Jul 28 18:34:02.316889 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.316793 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" Jul 28 18:34:02.458238 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.458199 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz"] Jul 28 18:34:02.459165 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:34:02.459143 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a101bf_a426_401a_9a56_08b34eb1cc5a.slice/crio-db5e1cf4ac96ecd1a76187cd7eaca7fa21e443eba99649e013a26634378e5cf8 WatchSource:0}: Error finding container db5e1cf4ac96ecd1a76187cd7eaca7fa21e443eba99649e013a26634378e5cf8: Status 404 returned error can't find the container with id db5e1cf4ac96ecd1a76187cd7eaca7fa21e443eba99649e013a26634378e5cf8 Jul 28 18:34:02.692126 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:02.692087 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" event={"ID":"03a101bf-a426-401a-9a56-08b34eb1cc5a","Type":"ContainerStarted","Data":"db5e1cf4ac96ecd1a76187cd7eaca7fa21e443eba99649e013a26634378e5cf8"} Jul 28 18:34:03.699371 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:03.699334 2569 generic.go:356] "Generic (PLEG): container finished" podID="03a101bf-a426-401a-9a56-08b34eb1cc5a" containerID="05a8c30f4f614ff92ffdbe633297796099e25e1b23ecf2a6f15222606db3bec6" exitCode=0 Jul 28 18:34:03.699774 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:03.699419 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" event={"ID":"03a101bf-a426-401a-9a56-08b34eb1cc5a","Type":"ContainerDied","Data":"05a8c30f4f614ff92ffdbe633297796099e25e1b23ecf2a6f15222606db3bec6"} Jul 28 18:34:04.832404 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:04.832380 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" Jul 28 18:34:04.898655 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:04.898622 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a101bf-a426-401a-9a56-08b34eb1cc5a-maas-api-service-ca\") pod \"03a101bf-a426-401a-9a56-08b34eb1cc5a\" (UID: \"03a101bf-a426-401a-9a56-08b34eb1cc5a\") " Jul 28 18:34:04.898987 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:04.898965 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a101bf-a426-401a-9a56-08b34eb1cc5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03a101bf-a426-401a-9a56-08b34eb1cc5a" (UID: "03a101bf-a426-401a-9a56-08b34eb1cc5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:34:04.999377 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:04.999293 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a101bf-a426-401a-9a56-08b34eb1cc5a-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:34:05.711926 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:05.711886 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" Jul 28 18:34:05.712101 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:05.711904 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz" event={"ID":"03a101bf-a426-401a-9a56-08b34eb1cc5a","Type":"ContainerDied","Data":"db5e1cf4ac96ecd1a76187cd7eaca7fa21e443eba99649e013a26634378e5cf8"} Jul 28 18:34:05.712101 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:05.712018 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db5e1cf4ac96ecd1a76187cd7eaca7fa21e443eba99649e013a26634378e5cf8" Jul 28 18:34:12.527865 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:12.527830 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz"] Jul 28 18:34:12.530011 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:12.529990 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fec3f09a-kd2pz"] Jul 28 18:34:13.976909 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:34:13.976869 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a101bf-a426-401a-9a56-08b34eb1cc5a" path="/var/lib/kubelet/pods/03a101bf-a426-401a-9a56-08b34eb1cc5a/volumes" Jul 28 18:35:46.760248 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:46.760193 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm"] Jul 28 18:35:46.760674 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:46.760622 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a101bf-a426-401a-9a56-08b34eb1cc5a" containerName="revoke-keys" Jul 28 18:35:46.760674 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:46.760634 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a101bf-a426-401a-9a56-08b34eb1cc5a" containerName="revoke-keys" Jul 28 18:35:46.760750 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:46.760712 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a101bf-a426-401a-9a56-08b34eb1cc5a" containerName="revoke-keys" Jul 28 18:35:46.763735 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:46.763719 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" Jul 28 18:35:46.766209 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:46.766187 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:35:46.771933 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:46.771424 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm"] Jul 28 18:35:46.936325 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:46.936290 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c78247-89c3-4ae8-93c8-6c7811c7d592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm\" (UID: \"c4c78247-89c3-4ae8-93c8-6c7811c7d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" Jul 28 18:35:47.037094 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:47.037008 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c78247-89c3-4ae8-93c8-6c7811c7d592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm\" (UID: \"c4c78247-89c3-4ae8-93c8-6c7811c7d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" Jul 28 18:35:47.037665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:47.037647 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c78247-89c3-4ae8-93c8-6c7811c7d592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm\" (UID: \"c4c78247-89c3-4ae8-93c8-6c7811c7d592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" Jul 28 18:35:47.077220 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:47.077174 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" Jul 28 18:35:47.214323 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:47.214295 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm"] Jul 28 18:35:47.215703 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:35:47.215677 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c78247_89c3_4ae8_93c8_6c7811c7d592.slice/crio-c43b06539c759595a836862c33afa24b14f13fa9c2626aa9ba1956f7759d4fd9 WatchSource:0}: Error finding container c43b06539c759595a836862c33afa24b14f13fa9c2626aa9ba1956f7759d4fd9: Status 404 returned error can't find the container with id c43b06539c759595a836862c33afa24b14f13fa9c2626aa9ba1956f7759d4fd9 Jul 28 18:35:47.279071 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:47.279042 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" event={"ID":"c4c78247-89c3-4ae8-93c8-6c7811c7d592","Type":"ContainerStarted","Data":"c43b06539c759595a836862c33afa24b14f13fa9c2626aa9ba1956f7759d4fd9"} Jul 28 18:35:48.285888 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:48.285797 2569 generic.go:356] "Generic (PLEG): container finished" podID="c4c78247-89c3-4ae8-93c8-6c7811c7d592" containerID="8da6c039a9abb617cf74739c2d05eab8e0d05351cf8e4c2d3b39f440b71ccd08" exitCode=0 Jul 28 18:35:48.286293 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:48.285884 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" event={"ID":"c4c78247-89c3-4ae8-93c8-6c7811c7d592","Type":"ContainerDied","Data":"8da6c039a9abb617cf74739c2d05eab8e0d05351cf8e4c2d3b39f440b71ccd08"} Jul 28 18:35:49.426727 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:49.426704 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" Jul 28 18:35:49.559611 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:49.559526 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c78247-89c3-4ae8-93c8-6c7811c7d592-maas-api-service-ca\") pod \"c4c78247-89c3-4ae8-93c8-6c7811c7d592\" (UID: \"c4c78247-89c3-4ae8-93c8-6c7811c7d592\") " Jul 28 18:35:49.559883 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:49.559859 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c78247-89c3-4ae8-93c8-6c7811c7d592-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4c78247-89c3-4ae8-93c8-6c7811c7d592" (UID: "c4c78247-89c3-4ae8-93c8-6c7811c7d592"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:35:49.660789 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:49.660743 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c78247-89c3-4ae8-93c8-6c7811c7d592-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:35:50.297910 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:50.297835 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" Jul 28 18:35:50.297910 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:50.297876 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm" event={"ID":"c4c78247-89c3-4ae8-93c8-6c7811c7d592","Type":"ContainerDied","Data":"c43b06539c759595a836862c33afa24b14f13fa9c2626aa9ba1956f7759d4fd9"} Jul 28 18:35:50.298095 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:50.297929 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c43b06539c759595a836862c33afa24b14f13fa9c2626aa9ba1956f7759d4fd9" Jul 28 18:35:56.052327 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:56.052296 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:35:56.057883 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:35:56.057854 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:36:02.304069 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:36:02.304032 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm"] Jul 28 18:36:02.306250 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:36:02.306213 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cfa56f3-4jnvm"] Jul 28 18:36:03.976333 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:36:03.976298 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4c78247-89c3-4ae8-93c8-6c7811c7d592" path="/var/lib/kubelet/pods/c4c78247-89c3-4ae8-93c8-6c7811c7d592/volumes" Jul 28 18:36:56.079547 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:36:56.079473 2569 scope.go:117] "RemoveContainer" containerID="49c243e334a9c993d8d492b9f373cff7cdb0eb68e8dc203bc646bfb3fc326800" Jul 28 18:37:56.103135 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:37:56.103102 2569 scope.go:117] "RemoveContainer" containerID="bf0b91bc8abc9f779884ef2c2e65d89992ced135afee7d5f476d52fd787d49fb" Jul 28 18:37:56.111816 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:37:56.111795 2569 scope.go:117] "RemoveContainer" containerID="b16cbdccb39e8e384328bdfbd97250acfe14a730e8b64a0e7881754d3ccb9e92" Jul 28 18:40:10.979516 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:10.979437 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z"] Jul 28 18:40:10.981842 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:10.979955 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c78247-89c3-4ae8-93c8-6c7811c7d592" containerName="revoke-keys" Jul 28 18:40:10.981842 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:10.979972 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c78247-89c3-4ae8-93c8-6c7811c7d592" containerName="revoke-keys" Jul 28 18:40:10.981842 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:10.980050 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c78247-89c3-4ae8-93c8-6c7811c7d592" containerName="revoke-keys" Jul 28 18:40:10.983005 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:10.982990 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" Jul 28 18:40:10.985654 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:10.985634 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:40:10.990762 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:10.990740 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z"] Jul 28 18:40:11.045196 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:11.045160 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347d034-7631-4347-afef-a508e0fb45d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z\" (UID: \"0347d034-7631-4347-afef-a508e0fb45d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" Jul 28 18:40:11.146735 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:11.146698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347d034-7631-4347-afef-a508e0fb45d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z\" (UID: \"0347d034-7631-4347-afef-a508e0fb45d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" Jul 28 18:40:11.147362 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:11.147343 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347d034-7631-4347-afef-a508e0fb45d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z\" (UID: \"0347d034-7631-4347-afef-a508e0fb45d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" Jul 28 18:40:11.294586 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:11.294508 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" Jul 28 18:40:11.434951 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:11.434923 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z"] Jul 28 18:40:11.436484 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:40:11.436447 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0347d034_7631_4347_afef_a508e0fb45d1.slice/crio-6f5a9297a4be270632bd7708e90cd8bfa77b63f8b173c64ed3ed320d6b7aa227 WatchSource:0}: Error finding container 6f5a9297a4be270632bd7708e90cd8bfa77b63f8b173c64ed3ed320d6b7aa227: Status 404 returned error can't find the container with id 6f5a9297a4be270632bd7708e90cd8bfa77b63f8b173c64ed3ed320d6b7aa227 Jul 28 18:40:11.793583 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:11.793542 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" event={"ID":"0347d034-7631-4347-afef-a508e0fb45d1","Type":"ContainerStarted","Data":"87aef506471ff726004fefb4ee9fa3a2de2d03079a4307ee3c3cc6876f36c82b"} Jul 28 18:40:11.793760 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:11.793610 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" event={"ID":"0347d034-7631-4347-afef-a508e0fb45d1","Type":"ContainerStarted","Data":"6f5a9297a4be270632bd7708e90cd8bfa77b63f8b173c64ed3ed320d6b7aa227"} Jul 28 18:40:11.810345 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:11.810297 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" podStartSLOduration=1.8102809949999998 podStartE2EDuration="1.810280995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:11.808970323 +0000 UTC m=+2356.422254733" watchObservedRunningTime="2026-07-28 18:40:11.810280995 +0000 UTC m=+2356.423565405" Jul 28 18:40:12.801399 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:12.801366 2569 generic.go:356] "Generic (PLEG): container finished" podID="0347d034-7631-4347-afef-a508e0fb45d1" containerID="87aef506471ff726004fefb4ee9fa3a2de2d03079a4307ee3c3cc6876f36c82b" exitCode=0 Jul 28 18:40:12.801810 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:12.801435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" event={"ID":"0347d034-7631-4347-afef-a508e0fb45d1","Type":"ContainerDied","Data":"87aef506471ff726004fefb4ee9fa3a2de2d03079a4307ee3c3cc6876f36c82b"} Jul 28 18:40:13.946586 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:13.946542 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" Jul 28 18:40:14.074692 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:14.074618 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347d034-7631-4347-afef-a508e0fb45d1-maas-api-service-ca\") pod \"0347d034-7631-4347-afef-a508e0fb45d1\" (UID: \"0347d034-7631-4347-afef-a508e0fb45d1\") " Jul 28 18:40:14.074952 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:14.074930 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0347d034-7631-4347-afef-a508e0fb45d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0347d034-7631-4347-afef-a508e0fb45d1" (UID: "0347d034-7631-4347-afef-a508e0fb45d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:40:14.175397 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:14.175366 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347d034-7631-4347-afef-a508e0fb45d1-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.814071 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:14.814041 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" Jul 28 18:40:14.814270 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:14.814077 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z" event={"ID":"0347d034-7631-4347-afef-a508e0fb45d1","Type":"ContainerDied","Data":"6f5a9297a4be270632bd7708e90cd8bfa77b63f8b173c64ed3ed320d6b7aa227"} Jul 28 18:40:14.814270 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:14.814136 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f5a9297a4be270632bd7708e90cd8bfa77b63f8b173c64ed3ed320d6b7aa227" Jul 28 18:40:21.620888 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:21.620850 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z"] Jul 28 18:40:21.623162 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:21.623138 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61d29d-rdv6z"] Jul 28 18:40:21.976344 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:21.976267 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0347d034-7631-4347-afef-a508e0fb45d1" path="/var/lib/kubelet/pods/0347d034-7631-4347-afef-a508e0fb45d1/volumes" Jul 28 18:40:28.809724 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:28.809695 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6"] Jul 28 18:40:28.810117 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:28.810104 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0347d034-7631-4347-afef-a508e0fb45d1" containerName="revoke-keys" Jul 28 18:40:28.810117 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:28.810118 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0347d034-7631-4347-afef-a508e0fb45d1" containerName="revoke-keys" Jul 28 18:40:28.810210 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:28.810190 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0347d034-7631-4347-afef-a508e0fb45d1" containerName="revoke-keys" Jul 28 18:40:28.813493 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:28.813479 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" Jul 28 18:40:28.816215 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:28.816200 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:40:28.821222 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:28.821066 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6"] Jul 28 18:40:28.910035 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:28.909999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151f2fd8-7d5d-4103-8c68-24919575a7c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6\" (UID: \"151f2fd8-7d5d-4103-8c68-24919575a7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" Jul 28 18:40:29.011253 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:29.011200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151f2fd8-7d5d-4103-8c68-24919575a7c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6\" (UID: \"151f2fd8-7d5d-4103-8c68-24919575a7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" Jul 28 18:40:29.011816 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:29.011796 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151f2fd8-7d5d-4103-8c68-24919575a7c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6\" (UID: \"151f2fd8-7d5d-4103-8c68-24919575a7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" Jul 28 18:40:29.125310 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:29.125278 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" Jul 28 18:40:29.264975 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:29.264944 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6"] Jul 28 18:40:29.266187 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:40:29.266163 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151f2fd8_7d5d_4103_8c68_24919575a7c5.slice/crio-bb660c7b276b559e7d8a7fd5240635acb929c75b9a336c0b545930f507fd05ea WatchSource:0}: Error finding container bb660c7b276b559e7d8a7fd5240635acb929c75b9a336c0b545930f507fd05ea: Status 404 returned error can't find the container with id bb660c7b276b559e7d8a7fd5240635acb929c75b9a336c0b545930f507fd05ea Jul 28 18:40:29.907088 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:29.907058 2569 generic.go:356] "Generic (PLEG): container finished" podID="151f2fd8-7d5d-4103-8c68-24919575a7c5" containerID="2136b5d9c1b7a402346c544833bbce00d2d89f6185318ed76040d54fbff6e213" exitCode=0 Jul 28 18:40:29.907454 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:29.907105 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" event={"ID":"151f2fd8-7d5d-4103-8c68-24919575a7c5","Type":"ContainerDied","Data":"2136b5d9c1b7a402346c544833bbce00d2d89f6185318ed76040d54fbff6e213"} Jul 28 18:40:29.907454 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:29.907148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" event={"ID":"151f2fd8-7d5d-4103-8c68-24919575a7c5","Type":"ContainerStarted","Data":"bb660c7b276b559e7d8a7fd5240635acb929c75b9a336c0b545930f507fd05ea"} Jul 28 18:40:31.052645 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:31.052620 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" Jul 28 18:40:31.233949 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:31.233865 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151f2fd8-7d5d-4103-8c68-24919575a7c5-maas-api-service-ca\") pod \"151f2fd8-7d5d-4103-8c68-24919575a7c5\" (UID: \"151f2fd8-7d5d-4103-8c68-24919575a7c5\") " Jul 28 18:40:31.234328 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:31.234304 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151f2fd8-7d5d-4103-8c68-24919575a7c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "151f2fd8-7d5d-4103-8c68-24919575a7c5" (UID: "151f2fd8-7d5d-4103-8c68-24919575a7c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:40:31.334995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:31.334966 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/151f2fd8-7d5d-4103-8c68-24919575a7c5-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:40:31.921272 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:31.921246 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" Jul 28 18:40:31.921272 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:31.921255 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6" event={"ID":"151f2fd8-7d5d-4103-8c68-24919575a7c5","Type":"ContainerDied","Data":"bb660c7b276b559e7d8a7fd5240635acb929c75b9a336c0b545930f507fd05ea"} Jul 28 18:40:31.921484 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:31.921332 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb660c7b276b559e7d8a7fd5240635acb929c75b9a336c0b545930f507fd05ea" Jul 28 18:40:39.489097 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:39.489056 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6"] Jul 28 18:40:39.491377 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:39.491353 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db5d01-4mvn6"] Jul 28 18:40:39.976273 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:39.976215 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="151f2fd8-7d5d-4103-8c68-24919575a7c5" path="/var/lib/kubelet/pods/151f2fd8-7d5d-4103-8c68-24919575a7c5/volumes" Jul 28 18:40:56.089139 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:56.089110 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:40:56.097322 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:56.097303 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:40:56.152910 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:40:56.152883 2569 scope.go:117] "RemoveContainer" containerID="05a8c30f4f614ff92ffdbe633297796099e25e1b23ecf2a6f15222606db3bec6" Jul 28 18:41:47.097775 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.097742 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7"] Jul 28 18:41:47.098205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.098161 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151f2fd8-7d5d-4103-8c68-24919575a7c5" containerName="revoke-keys" Jul 28 18:41:47.098205 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.098172 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="151f2fd8-7d5d-4103-8c68-24919575a7c5" containerName="revoke-keys" Jul 28 18:41:47.098307 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.098266 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="151f2fd8-7d5d-4103-8c68-24919575a7c5" containerName="revoke-keys" Jul 28 18:41:47.101332 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.101316 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" Jul 28 18:41:47.104000 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.103981 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:41:47.110184 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.110160 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7"] Jul 28 18:41:47.177753 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.177716 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06905f23-d1dc-4658-967d-e70ddd85f5c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7\" (UID: \"06905f23-d1dc-4658-967d-e70ddd85f5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" Jul 28 18:41:47.278649 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.278598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06905f23-d1dc-4658-967d-e70ddd85f5c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7\" (UID: \"06905f23-d1dc-4658-967d-e70ddd85f5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" Jul 28 18:41:47.279289 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.279265 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06905f23-d1dc-4658-967d-e70ddd85f5c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7\" (UID: \"06905f23-d1dc-4658-967d-e70ddd85f5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" Jul 28 18:41:47.414207 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.414123 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" Jul 28 18:41:47.559134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:47.559104 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7"] Jul 28 18:41:47.560426 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:41:47.560398 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06905f23_d1dc_4658_967d_e70ddd85f5c5.slice/crio-2e0bfe676ca0b8eb2ed419c44af84d4b719589bfadca5520c229774fa396e2f4 WatchSource:0}: Error finding container 2e0bfe676ca0b8eb2ed419c44af84d4b719589bfadca5520c229774fa396e2f4: Status 404 returned error can't find the container with id 2e0bfe676ca0b8eb2ed419c44af84d4b719589bfadca5520c229774fa396e2f4 Jul 28 18:41:48.349813 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:48.349779 2569 generic.go:356] "Generic (PLEG): container finished" podID="06905f23-d1dc-4658-967d-e70ddd85f5c5" containerID="81173d34df57b276c6fc1f5d818e4ed855c7bf76db7dd02044f893c85cfc4062" exitCode=0 Jul 28 18:41:48.350181 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:48.349830 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" event={"ID":"06905f23-d1dc-4658-967d-e70ddd85f5c5","Type":"ContainerDied","Data":"81173d34df57b276c6fc1f5d818e4ed855c7bf76db7dd02044f893c85cfc4062"} Jul 28 18:41:48.350181 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:48.349874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" event={"ID":"06905f23-d1dc-4658-967d-e70ddd85f5c5","Type":"ContainerStarted","Data":"2e0bfe676ca0b8eb2ed419c44af84d4b719589bfadca5520c229774fa396e2f4"} Jul 28 18:41:49.493790 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:49.493763 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" Jul 28 18:41:49.600210 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:49.600183 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06905f23-d1dc-4658-967d-e70ddd85f5c5-maas-api-service-ca\") pod \"06905f23-d1dc-4658-967d-e70ddd85f5c5\" (UID: \"06905f23-d1dc-4658-967d-e70ddd85f5c5\") " Jul 28 18:41:49.600541 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:49.600520 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06905f23-d1dc-4658-967d-e70ddd85f5c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06905f23-d1dc-4658-967d-e70ddd85f5c5" (UID: "06905f23-d1dc-4658-967d-e70ddd85f5c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:41:49.701269 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:49.701163 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06905f23-d1dc-4658-967d-e70ddd85f5c5-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:41:50.362080 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:50.362053 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" Jul 28 18:41:50.362277 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:50.362057 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7" event={"ID":"06905f23-d1dc-4658-967d-e70ddd85f5c5","Type":"ContainerDied","Data":"2e0bfe676ca0b8eb2ed419c44af84d4b719589bfadca5520c229774fa396e2f4"} Jul 28 18:41:50.362277 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:50.362183 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e0bfe676ca0b8eb2ed419c44af84d4b719589bfadca5520c229774fa396e2f4" Jul 28 18:41:56.175811 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:41:56.175780 2569 scope.go:117] "RemoveContainer" containerID="8da6c039a9abb617cf74739c2d05eab8e0d05351cf8e4c2d3b39f440b71ccd08" Jul 28 18:42:02.634603 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:02.634563 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7"] Jul 28 18:42:02.636998 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:02.636970 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac88b3-tppw7"] Jul 28 18:42:03.976585 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:03.976552 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06905f23-d1dc-4658-967d-e70ddd85f5c5" path="/var/lib/kubelet/pods/06905f23-d1dc-4658-967d-e70ddd85f5c5/volumes" Jul 28 18:42:09.535368 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.535337 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5"] Jul 28 18:42:09.535841 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.535735 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06905f23-d1dc-4658-967d-e70ddd85f5c5" containerName="revoke-keys" Jul 28 18:42:09.535841 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.535746 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06905f23-d1dc-4658-967d-e70ddd85f5c5" containerName="revoke-keys" Jul 28 18:42:09.535841 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.535827 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="06905f23-d1dc-4658-967d-e70ddd85f5c5" containerName="revoke-keys" Jul 28 18:42:09.539103 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.539087 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" Jul 28 18:42:09.541617 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.541593 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:42:09.547070 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.547034 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5"] Jul 28 18:42:09.695807 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.695773 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ef14d5-595c-4b86-b789-e78c2c4462ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5\" (UID: \"15ef14d5-595c-4b86-b789-e78c2c4462ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" Jul 28 18:42:09.796944 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.796850 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ef14d5-595c-4b86-b789-e78c2c4462ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5\" (UID: \"15ef14d5-595c-4b86-b789-e78c2c4462ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" Jul 28 18:42:09.797507 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.797486 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ef14d5-595c-4b86-b789-e78c2c4462ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5\" (UID: \"15ef14d5-595c-4b86-b789-e78c2c4462ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" Jul 28 18:42:09.852428 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.852388 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" Jul 28 18:42:09.990150 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:09.990066 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5"] Jul 28 18:42:10.472917 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:10.472810 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" event={"ID":"15ef14d5-595c-4b86-b789-e78c2c4462ef","Type":"ContainerStarted","Data":"5f0b97db6e22d9183db1b766073867e2ee7b059bcf1e068cfa762a743668f524"} Jul 28 18:42:10.472917 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:10.472867 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" event={"ID":"15ef14d5-595c-4b86-b789-e78c2c4462ef","Type":"ContainerStarted","Data":"2e8e3f6429051a0848d5ea7aa63b3b3dbb58996af3b87f54e0aaf60d6e2fe4d8"} Jul 28 18:42:10.488404 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:10.488360 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" podStartSLOduration=1.488343491 podStartE2EDuration="1.488343491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:42:10.48794493 +0000 UTC m=+2475.101229361" watchObservedRunningTime="2026-07-28 18:42:10.488343491 +0000 UTC m=+2475.101627900" Jul 28 18:42:11.480742 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:11.480704 2569 generic.go:356] "Generic (PLEG): container finished" podID="15ef14d5-595c-4b86-b789-e78c2c4462ef" containerID="5f0b97db6e22d9183db1b766073867e2ee7b059bcf1e068cfa762a743668f524" exitCode=0 Jul 28 18:42:11.481123 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:11.480785 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" event={"ID":"15ef14d5-595c-4b86-b789-e78c2c4462ef","Type":"ContainerDied","Data":"5f0b97db6e22d9183db1b766073867e2ee7b059bcf1e068cfa762a743668f524"} Jul 28 18:42:12.621165 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:12.621140 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" Jul 28 18:42:12.721676 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:12.721639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ef14d5-595c-4b86-b789-e78c2c4462ef-maas-api-service-ca\") pod \"15ef14d5-595c-4b86-b789-e78c2c4462ef\" (UID: \"15ef14d5-595c-4b86-b789-e78c2c4462ef\") " Jul 28 18:42:12.722021 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:12.721996 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ef14d5-595c-4b86-b789-e78c2c4462ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15ef14d5-595c-4b86-b789-e78c2c4462ef" (UID: "15ef14d5-595c-4b86-b789-e78c2c4462ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:42:12.823047 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:12.822955 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ef14d5-595c-4b86-b789-e78c2c4462ef-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:42:13.493098 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:13.493071 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" Jul 28 18:42:13.493299 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:13.493069 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5" event={"ID":"15ef14d5-595c-4b86-b789-e78c2c4462ef","Type":"ContainerDied","Data":"2e8e3f6429051a0848d5ea7aa63b3b3dbb58996af3b87f54e0aaf60d6e2fe4d8"} Jul 28 18:42:13.493299 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:13.493190 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8e3f6429051a0848d5ea7aa63b3b3dbb58996af3b87f54e0aaf60d6e2fe4d8" Jul 28 18:42:25.266670 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:25.266634 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5"] Jul 28 18:42:25.269243 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:25.269202 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f7f010-kqmt5"] Jul 28 18:42:25.976559 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:42:25.976526 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ef14d5-595c-4b86-b789-e78c2c4462ef" path="/var/lib/kubelet/pods/15ef14d5-595c-4b86-b789-e78c2c4462ef/volumes" Jul 28 18:43:04.528113 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.528024 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx"] Jul 28 18:43:04.528947 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.528487 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ef14d5-595c-4b86-b789-e78c2c4462ef" containerName="revoke-keys" Jul 28 18:43:04.528947 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.528501 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ef14d5-595c-4b86-b789-e78c2c4462ef" containerName="revoke-keys" Jul 28 18:43:04.528947 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.528590 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ef14d5-595c-4b86-b789-e78c2c4462ef" containerName="revoke-keys" Jul 28 18:43:04.531731 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.531716 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:04.534346 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.534320 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-f36101\"" Jul 28 18:43:04.534465 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.534404 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-f36101-dockercfg-6wzkl\"" Jul 28 18:43:04.542184 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.542159 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx"] Jul 28 18:43:04.593652 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.593616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6z2\" (UniqueName: \"kubernetes.io/projected/e48a897c-f4cb-4362-b4f9-460d2dca4128-kube-api-access-kr6z2\") pod \"payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx\" (UID: \"e48a897c-f4cb-4362-b4f9-460d2dca4128\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:04.593818 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.593660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48a897c-f4cb-4362-b4f9-460d2dca4128-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx\" (UID: \"e48a897c-f4cb-4362-b4f9-460d2dca4128\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:04.694429 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.694392 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6z2\" (UniqueName: \"kubernetes.io/projected/e48a897c-f4cb-4362-b4f9-460d2dca4128-kube-api-access-kr6z2\") pod \"payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx\" (UID: \"e48a897c-f4cb-4362-b4f9-460d2dca4128\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:04.694429 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.694436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48a897c-f4cb-4362-b4f9-460d2dca4128-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx\" (UID: \"e48a897c-f4cb-4362-b4f9-460d2dca4128\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:04.695072 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.695052 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48a897c-f4cb-4362-b4f9-460d2dca4128-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx\" (UID: \"e48a897c-f4cb-4362-b4f9-460d2dca4128\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:04.702338 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.702308 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6z2\" (UniqueName: \"kubernetes.io/projected/e48a897c-f4cb-4362-b4f9-460d2dca4128-kube-api-access-kr6z2\") pod \"payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx\" (UID: \"e48a897c-f4cb-4362-b4f9-460d2dca4128\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:04.844063 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.844028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:04.983586 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:04.983559 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx"] Jul 28 18:43:04.984950 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:43:04.984924 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48a897c_f4cb_4362_b4f9_460d2dca4128.slice/crio-687fae681c3ceef24a71f0f6ff1b61450618c9463dc883b65024a3e0fd3420e8 WatchSource:0}: Error finding container 687fae681c3ceef24a71f0f6ff1b61450618c9463dc883b65024a3e0fd3420e8: Status 404 returned error can't find the container with id 687fae681c3ceef24a71f0f6ff1b61450618c9463dc883b65024a3e0fd3420e8 Jul 28 18:43:05.795639 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:05.795605 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" event={"ID":"e48a897c-f4cb-4362-b4f9-460d2dca4128","Type":"ContainerStarted","Data":"8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46"} Jul 28 18:43:05.796032 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:05.795657 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" event={"ID":"e48a897c-f4cb-4362-b4f9-460d2dca4128","Type":"ContainerStarted","Data":"687fae681c3ceef24a71f0f6ff1b61450618c9463dc883b65024a3e0fd3420e8"} Jul 28 18:43:05.796032 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:05.795732 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:43:05.815252 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:05.815195 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" podStartSLOduration=1.815179484 podStartE2EDuration="1.815179484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:43:05.812740048 +0000 UTC m=+2530.426024479" watchObservedRunningTime="2026-07-28 18:43:05.815179484 +0000 UTC m=+2530.428463884" Jul 28 18:43:16.803442 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:43:16.803408 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:45:00.133588 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.133552 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns"] Jul 28 18:45:00.137502 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.137480 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7"] Jul 28 18:45:00.137656 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.137612 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" Jul 28 18:45:00.140111 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.140083 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:45:00.140768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.140748 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m"] Jul 28 18:45:00.140905 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.140886 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7" Jul 28 18:45:00.149033 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.149008 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns"] Jul 28 18:45:00.149299 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.149283 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m" Jul 28 18:45:00.150333 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.149839 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m"] Jul 28 18:45:00.152162 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.152139 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" Jul 28 18:45:00.153158 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.153130 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7"] Jul 28 18:45:00.161896 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.161875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7" Jul 28 18:45:00.197789 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.197765 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m" Jul 28 18:45:00.372720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.372696 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m"] Jul 28 18:45:00.373716 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:45:00.373694 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903afd21_dbb2_4e0f_85d1_59fb138a34b8.slice/crio-83cf85c5d6a218e8cb490be58fd764194447d7e707c08f7a509474ef4933b198 WatchSource:0}: Error finding container 83cf85c5d6a218e8cb490be58fd764194447d7e707c08f7a509474ef4933b198: Status 404 returned error can't find the container with id 83cf85c5d6a218e8cb490be58fd764194447d7e707c08f7a509474ef4933b198 Jul 28 18:45:00.455654 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.455622 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m" event={"ID":"903afd21-dbb2-4e0f-85d1-59fb138a34b8","Type":"ContainerStarted","Data":"83cf85c5d6a218e8cb490be58fd764194447d7e707c08f7a509474ef4933b198"} Jul 28 18:45:00.529478 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:45:00.529450 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109d4a5b_527a_409b_94b4_8318705764da.slice/crio-51c0ba06a67c800ba98871ff911b9dfba6bfc338268ab5ae17a060530aa334f1 WatchSource:0}: Error finding container 51c0ba06a67c800ba98871ff911b9dfba6bfc338268ab5ae17a060530aa334f1: Status 404 returned error can't find the container with id 51c0ba06a67c800ba98871ff911b9dfba6bfc338268ab5ae17a060530aa334f1 Jul 28 18:45:00.529662 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.529636 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns"] Jul 28 18:45:00.554278 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:00.554251 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7"] Jul 28 18:45:00.556683 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:45:00.556660 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18131847_6944_4c99_b81c_3c0fa8e6434c.slice/crio-c8b895fafffabe99e4a34278fa9e4de08067057891d0ebe30ee4c0457d365e18 WatchSource:0}: Error finding container c8b895fafffabe99e4a34278fa9e4de08067057891d0ebe30ee4c0457d365e18: Status 404 returned error can't find the container with id c8b895fafffabe99e4a34278fa9e4de08067057891d0ebe30ee4c0457d365e18 Jul 28 18:45:01.470689 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:01.470655 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" event={"ID":"109d4a5b-527a-409b-94b4-8318705764da","Type":"ContainerStarted","Data":"dc979abc7dcc91e5e9b717d81508b5b5a96dde6e62f9e132d7d082bd9e89fe69"} Jul 28 18:45:01.471114 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:01.470744 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" event={"ID":"109d4a5b-527a-409b-94b4-8318705764da","Type":"ContainerStarted","Data":"51c0ba06a67c800ba98871ff911b9dfba6bfc338268ab5ae17a060530aa334f1"} Jul 28 18:45:01.472154 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:01.472131 2569 generic.go:356] "Generic (PLEG): container finished" podID="903afd21-dbb2-4e0f-85d1-59fb138a34b8" containerID="1b0068987c4b7d8ca295a9daa32b72f51a945c56e69a0f1a4240bd093ef5499c" exitCode=0 Jul 28 18:45:01.472282 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:01.472206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m" event={"ID":"903afd21-dbb2-4e0f-85d1-59fb138a34b8","Type":"ContainerDied","Data":"1b0068987c4b7d8ca295a9daa32b72f51a945c56e69a0f1a4240bd093ef5499c"} Jul 28 18:45:01.473619 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:01.473597 2569 generic.go:356] "Generic (PLEG): container finished" podID="18131847-6944-4c99-b81c-3c0fa8e6434c" containerID="67ec247db14569051a5724847806272def68e80cc2af42bd19aa64d1966b5481" exitCode=0 Jul 28 18:45:01.473695 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:01.473647 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7" event={"ID":"18131847-6944-4c99-b81c-3c0fa8e6434c","Type":"ContainerDied","Data":"67ec247db14569051a5724847806272def68e80cc2af42bd19aa64d1966b5481"} Jul 28 18:45:01.473695 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:01.473680 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7" event={"ID":"18131847-6944-4c99-b81c-3c0fa8e6434c","Type":"ContainerStarted","Data":"c8b895fafffabe99e4a34278fa9e4de08067057891d0ebe30ee4c0457d365e18"} Jul 28 18:45:01.488662 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:01.488556 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" podStartSLOduration=1.488536374 podStartE2EDuration="1.488536374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:45:01.487813431 +0000 UTC m=+2646.101097841" watchObservedRunningTime="2026-07-28 18:45:01.488536374 +0000 UTC m=+2646.101820787" Jul 28 18:45:02.481246 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:02.481196 2569 generic.go:356] "Generic (PLEG): container finished" podID="109d4a5b-527a-409b-94b4-8318705764da" containerID="dc979abc7dcc91e5e9b717d81508b5b5a96dde6e62f9e132d7d082bd9e89fe69" exitCode=0 Jul 28 18:45:02.481637 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:02.481272 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" event={"ID":"109d4a5b-527a-409b-94b4-8318705764da","Type":"ContainerDied","Data":"dc979abc7dcc91e5e9b717d81508b5b5a96dde6e62f9e132d7d082bd9e89fe69"} Jul 28 18:45:02.639648 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:02.639621 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7" Jul 28 18:45:02.661928 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:02.661899 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m" Jul 28 18:45:03.488920 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:03.488889 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m" Jul 28 18:45:03.489432 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:03.488882 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-qxz9m" event={"ID":"903afd21-dbb2-4e0f-85d1-59fb138a34b8","Type":"ContainerDied","Data":"83cf85c5d6a218e8cb490be58fd764194447d7e707c08f7a509474ef4933b198"} Jul 28 18:45:03.489432 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:03.489025 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83cf85c5d6a218e8cb490be58fd764194447d7e707c08f7a509474ef4933b198" Jul 28 18:45:03.490202 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:03.490185 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7" Jul 28 18:45:03.490324 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:03.490249 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7" event={"ID":"18131847-6944-4c99-b81c-3c0fa8e6434c","Type":"ContainerDied","Data":"c8b895fafffabe99e4a34278fa9e4de08067057891d0ebe30ee4c0457d365e18"} Jul 28 18:45:03.490324 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:03.490289 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b895fafffabe99e4a34278fa9e4de08067057891d0ebe30ee4c0457d365e18" Jul 28 18:45:03.634542 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:03.634516 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" Jul 28 18:45:04.497717 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:04.497686 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" Jul 28 18:45:04.498142 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:04.497715 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns" event={"ID":"109d4a5b-527a-409b-94b4-8318705764da","Type":"ContainerDied","Data":"51c0ba06a67c800ba98871ff911b9dfba6bfc338268ab5ae17a060530aa334f1"} Jul 28 18:45:04.498142 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:04.497761 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c0ba06a67c800ba98871ff911b9dfba6bfc338268ab5ae17a060530aa334f1" Jul 28 18:45:56.130274 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:56.130183 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:45:56.139179 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:45:56.139158 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:46:56.240924 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:46:56.240887 2569 scope.go:117] "RemoveContainer" containerID="2136b5d9c1b7a402346c544833bbce00d2d89f6185318ed76040d54fbff6e213" Jul 28 18:46:56.249686 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:46:56.249666 2569 scope.go:117] "RemoveContainer" containerID="87aef506471ff726004fefb4ee9fa3a2de2d03079a4307ee3c3cc6876f36c82b" Jul 28 18:47:56.274703 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:47:56.274671 2569 scope.go:117] "RemoveContainer" containerID="81173d34df57b276c6fc1f5d818e4ed855c7bf76db7dd02044f893c85cfc4062" Jul 28 18:48:56.300472 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:48:56.300401 2569 scope.go:117] "RemoveContainer" containerID="5f0b97db6e22d9183db1b766073867e2ee7b059bcf1e068cfa762a743668f524" Jul 28 18:50:56.166720 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:50:56.166693 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:50:56.178425 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:50:56.178404 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:51:50.638900 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.638827 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w"] Jul 28 18:51:50.639361 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639321 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903afd21-dbb2-4e0f-85d1-59fb138a34b8" containerName="cleanup" Jul 28 18:51:50.639361 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639335 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="903afd21-dbb2-4e0f-85d1-59fb138a34b8" containerName="cleanup" Jul 28 18:51:50.639361 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639345 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="109d4a5b-527a-409b-94b4-8318705764da" containerName="cleanup" Jul 28 18:51:50.639361 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639351 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="109d4a5b-527a-409b-94b4-8318705764da" containerName="cleanup" Jul 28 18:51:50.639361 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639363 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18131847-6944-4c99-b81c-3c0fa8e6434c" containerName="cleanup" Jul 28 18:51:50.639565 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639369 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="18131847-6944-4c99-b81c-3c0fa8e6434c" containerName="cleanup" Jul 28 18:51:50.639565 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639448 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="109d4a5b-527a-409b-94b4-8318705764da" containerName="cleanup" Jul 28 18:51:50.639565 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639457 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="903afd21-dbb2-4e0f-85d1-59fb138a34b8" containerName="cleanup" Jul 28 18:51:50.639565 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.639465 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="18131847-6944-4c99-b81c-3c0fa8e6434c" containerName="cleanup" Jul 28 18:51:50.642280 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.642263 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:50.644868 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.644842 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c1237e-serving-cert\"" Jul 28 18:51:50.658741 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.658705 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w"] Jul 28 18:51:50.705944 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.705915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:50.706097 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.706021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:50.706097 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.706077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hrm\" (UniqueName: \"kubernetes.io/projected/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-kube-api-access-47hrm\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:50.806873 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.806833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:50.807043 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.806894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hrm\" (UniqueName: \"kubernetes.io/projected/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-kube-api-access-47hrm\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:50.807043 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.806932 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:50.807043 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:51:50.806997 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-serving-cert: secret "maas-api-e2e-ipp-a-c1237e-serving-cert" not found Jul 28 18:51:50.807161 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:51:50.807075 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls podName:c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f nodeName:}" failed. No retries permitted until 2026-07-28 18:51:51.307054151 +0000 UTC m=+3055.920338553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls") pod "maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" (UID: "c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f") : secret "maas-api-e2e-ipp-a-c1237e-serving-cert" not found Jul 28 18:51:50.807560 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.807539 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:50.815312 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:50.815289 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hrm\" (UniqueName: \"kubernetes.io/projected/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-kube-api-access-47hrm\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:51.311276 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:51.311222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:51.313785 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:51.313758 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:51.554357 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:51.554319 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:51.696790 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:51.696759 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w"] Jul 28 18:51:51.698670 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:51:51.698628 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b97d82_08cd_449b_a7e7_3a3f3fc8f88f.slice/crio-cacab888a4528959f1a0277e1740c0e864d7a31eed1e59285c1cd7e7cb5d9eb3 WatchSource:0}: Error finding container cacab888a4528959f1a0277e1740c0e864d7a31eed1e59285c1cd7e7cb5d9eb3: Status 404 returned error can't find the container with id cacab888a4528959f1a0277e1740c0e864d7a31eed1e59285c1cd7e7cb5d9eb3 Jul 28 18:51:51.700258 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:51.700237 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:51:51.848628 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:51.848591 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" event={"ID":"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f","Type":"ContainerStarted","Data":"cacab888a4528959f1a0277e1740c0e864d7a31eed1e59285c1cd7e7cb5d9eb3"} Jul 28 18:51:52.856052 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:52.856015 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" event={"ID":"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f","Type":"ContainerStarted","Data":"e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3"} Jul 28 18:51:52.856572 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:52.856147 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:51:52.874176 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:52.874118 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" podStartSLOduration=2.300253234 podStartE2EDuration="2.874102087s" totalImagesPullingTime="573.848853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:51:52.872806112 +0000 UTC m=+3057.486090557" watchObservedRunningTime="2026-07-28 18:51:52.874102087 +0000 UTC m=+3057.487386535" Jul 28 18:51:58.868941 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:51:58.868901 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:52:16.454431 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.454398 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm"] Jul 28 18:52:16.458371 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.458350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.461017 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.460992 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0ae953-openshift-default-dockercfg-xf6mh\"" Jul 28 18:52:16.480698 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.480669 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm"] Jul 28 18:52:16.538673 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.538625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-credential-socket\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.538847 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.538689 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b1f347-bc64-404a-ac8a-85256aa0044c-istiod-ca-cert\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.538847 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.538785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-token\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.538971 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.538865 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-envoy\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.538971 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.538908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-data\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.538971 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.538937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-podinfo\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.539124 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.538993 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl26t\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-kube-api-access-sl26t\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.539124 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.539043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-socket\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.539124 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.539072 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-certs\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.639546 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.639515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-credential-socket\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.639546 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.639548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b1f347-bc64-404a-ac8a-85256aa0044c-istiod-ca-cert\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.639807 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.639573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-token\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.639807 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.639601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-envoy\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.639807 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.639635 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-data\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.639807 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.639741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-podinfo\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.640017 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.639968 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-credential-socket\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.640268 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.640243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl26t\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-kube-api-access-sl26t\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.640437 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.640418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-socket\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.640520 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.640344 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-data\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.640520 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.640452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-certs\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.640520 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.640357 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b1f347-bc64-404a-ac8a-85256aa0044c-istiod-ca-cert\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.640756 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.640730 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-socket\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.640756 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.640745 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-certs\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.642397 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.642374 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-podinfo\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.642505 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.642491 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-envoy\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.650967 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.650939 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-token\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.651128 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.651106 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl26t\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-kube-api-access-sl26t\") pod \"e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.780049 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.779970 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:16.922031 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.922001 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm"] Jul 28 18:52:16.923747 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:52:16.923718 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b1f347_bc64_404a_ac8a_85256aa0044c.slice/crio-1a867f9899f51480e99f0cf30bfd0123a1158a2e3995fac48373aa550f3a6d0f WatchSource:0}: Error finding container 1a867f9899f51480e99f0cf30bfd0123a1158a2e3995fac48373aa550f3a6d0f: Status 404 returned error can't find the container with id 1a867f9899f51480e99f0cf30bfd0123a1158a2e3995fac48373aa550f3a6d0f Jul 28 18:52:16.994446 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:16.994415 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" event={"ID":"92b1f347-bc64-404a-ac8a-85256aa0044c","Type":"ContainerStarted","Data":"1a867f9899f51480e99f0cf30bfd0123a1158a2e3995fac48373aa550f3a6d0f"} Jul 28 18:52:19.906159 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:19.906116 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 28 18:52:19.906474 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:19.906201 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 28 18:52:19.906474 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:19.906245 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 28 18:52:20.015434 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:20.015393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" event={"ID":"92b1f347-bc64-404a-ac8a-85256aa0044c","Type":"ContainerStarted","Data":"ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4"} Jul 28 18:52:20.038414 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:20.038362 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podStartSLOduration=1.058432092 podStartE2EDuration="4.038344793s" totalImagesPullingTime="2.979912701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:52:20.035317116 +0000 UTC m=+3084.648601526" watchObservedRunningTime="2026-07-28 18:52:20.038344793 +0000 UTC m=+3084.651629253" Jul 28 18:52:20.780407 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:20.780371 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:20.782158 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:20.782120 2569 patch_prober.go:28] interesting pod/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 28 18:52:20.782301 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:20.782181 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 28 18:52:21.784674 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:21.784584 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:22.036498 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:22.036378 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:22.037660 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:22.037640 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:22.888386 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:22.888357 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf"] Jul 28 18:52:22.892300 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:22.892281 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:22.895080 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:22.895057 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ae953\"" Jul 28 18:52:22.895211 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:22.895195 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ae953-dockercfg-vczq6\"" Jul 28 18:52:22.901847 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:22.901827 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf"] Jul 28 18:52:23.006147 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:23.006106 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd49n\" (UniqueName: \"kubernetes.io/projected/92d6c15a-38a4-494c-86a3-6229ea6de759-kube-api-access-fd49n\") pod \"payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf\" (UID: \"92d6c15a-38a4-494c-86a3-6229ea6de759\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:23.006352 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:23.006194 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d6c15a-38a4-494c-86a3-6229ea6de759-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf\" (UID: \"92d6c15a-38a4-494c-86a3-6229ea6de759\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:23.106945 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:23.106908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd49n\" (UniqueName: \"kubernetes.io/projected/92d6c15a-38a4-494c-86a3-6229ea6de759-kube-api-access-fd49n\") pod \"payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf\" (UID: \"92d6c15a-38a4-494c-86a3-6229ea6de759\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:23.107111 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:23.106994 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d6c15a-38a4-494c-86a3-6229ea6de759-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf\" (UID: \"92d6c15a-38a4-494c-86a3-6229ea6de759\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:23.107741 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:23.107724 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d6c15a-38a4-494c-86a3-6229ea6de759-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf\" (UID: \"92d6c15a-38a4-494c-86a3-6229ea6de759\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:23.114632 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:23.114609 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd49n\" (UniqueName: \"kubernetes.io/projected/92d6c15a-38a4-494c-86a3-6229ea6de759-kube-api-access-fd49n\") pod \"payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf\" (UID: \"92d6c15a-38a4-494c-86a3-6229ea6de759\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:23.206091 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:23.206011 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:23.359051 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:23.353581 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf"] Jul 28 18:52:23.364717 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:52:23.364678 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d6c15a_38a4_494c_86a3_6229ea6de759.slice/crio-9dfe2e7d098ccc7cb803efc8933f634c4cdc9d1fa4e8fc1297c25227566f2eaf WatchSource:0}: Error finding container 9dfe2e7d098ccc7cb803efc8933f634c4cdc9d1fa4e8fc1297c25227566f2eaf: Status 404 returned error can't find the container with id 9dfe2e7d098ccc7cb803efc8933f634c4cdc9d1fa4e8fc1297c25227566f2eaf Jul 28 18:52:24.050839 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:24.050794 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" event={"ID":"92d6c15a-38a4-494c-86a3-6229ea6de759","Type":"ContainerStarted","Data":"86223431046c2b9196e717c4a66eaabbc8a6b61c1d615496087422a88a230c43"} Jul 28 18:52:24.051340 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:24.050860 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" event={"ID":"92d6c15a-38a4-494c-86a3-6229ea6de759","Type":"ContainerStarted","Data":"9dfe2e7d098ccc7cb803efc8933f634c4cdc9d1fa4e8fc1297c25227566f2eaf"} Jul 28 18:52:24.051340 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:24.051019 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:24.071797 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:24.071729 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" podStartSLOduration=2.071713464 podStartE2EDuration="2.071713464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:52:24.069144501 +0000 UTC m=+3088.682428913" watchObservedRunningTime="2026-07-28 18:52:24.071713464 +0000 UTC m=+3088.684997874" Jul 28 18:52:35.058564 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:35.058531 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:52:37.038075 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:37.038037 2569 patch_prober.go:28] interesting pod/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 28 18:52:37.038552 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:37.038105 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 28 18:52:37.129507 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:37.129473 2569 generic.go:356] "Generic (PLEG): container finished" podID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerID="ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4" exitCode=0 Jul 28 18:52:37.129665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:37.129544 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" event={"ID":"92b1f347-bc64-404a-ac8a-85256aa0044c","Type":"ContainerDied","Data":"ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4"} Jul 28 18:52:37.129931 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:37.129918 2569 scope.go:117] "RemoveContainer" containerID="ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4" Jul 28 18:52:37.130969 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:37.130943 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 28 18:52:37.131023 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:37.131010 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 28 18:52:37.131060 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:37.131046 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 28 18:52:38.137604 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:38.137573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" event={"ID":"92b1f347-bc64-404a-ac8a-85256aa0044c","Type":"ContainerStarted","Data":"dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4"} Jul 28 18:52:38.780302 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:38.780271 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:38.780567 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:38.780544 2569 patch_prober.go:28] interesting pod/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 28 18:52:38.780620 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:38.780594 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 28 18:52:39.780933 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:39.780896 2569 patch_prober.go:28] interesting pod/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 28 18:52:39.781342 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:39.780953 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 28 18:52:40.780488 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:40.780444 2569 patch_prober.go:28] interesting pod/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 28 18:52:40.780675 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:40.780506 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 28 18:52:41.781011 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:41.780973 2569 patch_prober.go:28] interesting pod/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 28 18:52:41.781430 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:41.781050 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 28 18:52:42.785011 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:42.784977 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:42.785495 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:42.785339 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:42.786071 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:42.786045 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:52:53.871543 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:53.871514 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9kkbx_b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59/payload-processing/0.log" Jul 28 18:52:53.985929 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:52:53.985900 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf_92d6c15a-38a4-494c-86a3-6229ea6de759/payload-processing/0.log" Jul 28 18:53:29.110050 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.110009 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-55swh"] Jul 28 18:53:29.113787 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.113768 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:29.116251 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.116217 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-h2x82\"" Jul 28 18:53:29.121562 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.121538 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-55swh"] Jul 28 18:53:29.222062 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.222030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx29p\" (UniqueName: \"kubernetes.io/projected/2a6b3d68-71b7-488f-a414-42993e2e3a13-kube-api-access-fx29p\") pod \"kuadrant-operator-catalog-55swh\" (UID: \"2a6b3d68-71b7-488f-a414-42993e2e3a13\") " pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:29.323504 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.323474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx29p\" (UniqueName: \"kubernetes.io/projected/2a6b3d68-71b7-488f-a414-42993e2e3a13-kube-api-access-fx29p\") pod \"kuadrant-operator-catalog-55swh\" (UID: \"2a6b3d68-71b7-488f-a414-42993e2e3a13\") " pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:29.332034 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.332012 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx29p\" (UniqueName: \"kubernetes.io/projected/2a6b3d68-71b7-488f-a414-42993e2e3a13-kube-api-access-fx29p\") pod \"kuadrant-operator-catalog-55swh\" (UID: \"2a6b3d68-71b7-488f-a414-42993e2e3a13\") " pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:29.427981 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.427902 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:29.566576 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:29.566552 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-55swh"] Jul 28 18:53:29.568040 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:53:29.568008 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6b3d68_71b7_488f_a414_42993e2e3a13.slice/crio-a84c3a8932aada810336a48410a735d2a9ac1bb3a39ccfd4808a299999c2715b WatchSource:0}: Error finding container a84c3a8932aada810336a48410a735d2a9ac1bb3a39ccfd4808a299999c2715b: Status 404 returned error can't find the container with id a84c3a8932aada810336a48410a735d2a9ac1bb3a39ccfd4808a299999c2715b Jul 28 18:53:30.474701 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:30.474664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-55swh" event={"ID":"2a6b3d68-71b7-488f-a414-42993e2e3a13","Type":"ContainerStarted","Data":"a84c3a8932aada810336a48410a735d2a9ac1bb3a39ccfd4808a299999c2715b"} Jul 28 18:53:31.481917 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:31.481885 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-55swh" event={"ID":"2a6b3d68-71b7-488f-a414-42993e2e3a13","Type":"ContainerStarted","Data":"13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f"} Jul 28 18:53:31.502399 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:31.502297 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-55swh" podStartSLOduration=0.831843518 podStartE2EDuration="2.502280583s" totalImagesPullingTime="1.670437065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:31.500971638 +0000 UTC m=+3156.114256049" watchObservedRunningTime="2026-07-28 18:53:31.502280583 +0000 UTC m=+3156.115564993" Jul 28 18:53:38.881891 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:38.881860 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9kkbx_b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59/payload-processing/0.log" Jul 28 18:53:39.002402 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:39.002372 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf_92d6c15a-38a4-494c-86a3-6229ea6de759/payload-processing/0.log" Jul 28 18:53:39.428618 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:39.428585 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:39.428811 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:39.428665 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:39.450673 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:39.450646 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:39.560732 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:39.560701 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:39.679277 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:39.679183 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-55swh"] Jul 28 18:53:41.548563 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:41.548523 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-55swh" podUID="2a6b3d68-71b7-488f-a414-42993e2e3a13" containerName="registry-server" containerID="cri-o://13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f" gracePeriod=2 Jul 28 18:53:41.796149 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:41.796121 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:41.840480 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:41.840449 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx29p\" (UniqueName: \"kubernetes.io/projected/2a6b3d68-71b7-488f-a414-42993e2e3a13-kube-api-access-fx29p\") pod \"2a6b3d68-71b7-488f-a414-42993e2e3a13\" (UID: \"2a6b3d68-71b7-488f-a414-42993e2e3a13\") " Jul 28 18:53:41.842678 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:41.842637 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6b3d68-71b7-488f-a414-42993e2e3a13-kube-api-access-fx29p" (OuterVolumeSpecName: "kube-api-access-fx29p") pod "2a6b3d68-71b7-488f-a414-42993e2e3a13" (UID: "2a6b3d68-71b7-488f-a414-42993e2e3a13"). InnerVolumeSpecName "kube-api-access-fx29p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:53:41.941432 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:41.941395 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx29p\" (UniqueName: \"kubernetes.io/projected/2a6b3d68-71b7-488f-a414-42993e2e3a13-kube-api-access-fx29p\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:53:42.556354 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.556318 2569 generic.go:356] "Generic (PLEG): container finished" podID="2a6b3d68-71b7-488f-a414-42993e2e3a13" containerID="13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f" exitCode=0 Jul 28 18:53:42.556833 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.556362 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-55swh" Jul 28 18:53:42.556833 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.556408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-55swh" event={"ID":"2a6b3d68-71b7-488f-a414-42993e2e3a13","Type":"ContainerDied","Data":"13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f"} Jul 28 18:53:42.556833 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.556461 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-55swh" event={"ID":"2a6b3d68-71b7-488f-a414-42993e2e3a13","Type":"ContainerDied","Data":"a84c3a8932aada810336a48410a735d2a9ac1bb3a39ccfd4808a299999c2715b"} Jul 28 18:53:42.556833 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.556482 2569 scope.go:117] "RemoveContainer" containerID="13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f" Jul 28 18:53:42.568065 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.568043 2569 scope.go:117] "RemoveContainer" containerID="13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f" Jul 28 18:53:42.568376 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:53:42.568345 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f\": container with ID starting with 13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f not found: ID does not exist" containerID="13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f" Jul 28 18:53:42.568478 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.568385 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f"} err="failed to get container status \"13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f\": rpc error: code = NotFound desc = could not find container \"13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f\": container with ID starting with 13a51a10a52c089e57bb1a6ab2d09f7e43bdcc0f1419d8e50c21c4036ceed71f not found: ID does not exist" Jul 28 18:53:42.578820 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.578790 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-55swh"] Jul 28 18:53:42.586012 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.585987 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-55swh"] Jul 28 18:53:42.837119 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.837036 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x"] Jul 28 18:53:42.837500 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.837485 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a6b3d68-71b7-488f-a414-42993e2e3a13" containerName="registry-server" Jul 28 18:53:42.837560 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.837501 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a6b3d68-71b7-488f-a414-42993e2e3a13" containerName="registry-server" Jul 28 18:53:42.837603 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.837578 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a6b3d68-71b7-488f-a414-42993e2e3a13" containerName="registry-server" Jul 28 18:53:42.842031 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.842009 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:42.844670 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.844651 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-51b7e8\"" Jul 28 18:53:42.844779 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.844706 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-51b7e8-dockercfg-ckbwd\"" Jul 28 18:53:42.849971 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.849944 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x"] Jul 28 18:53:42.951310 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.951241 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8712b4-cc01-431f-91cd-c28f0a95b97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x\" (UID: \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:42.951524 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:42.951446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljsj\" (UniqueName: \"kubernetes.io/projected/6a8712b4-cc01-431f-91cd-c28f0a95b97c-kube-api-access-5ljsj\") pod \"payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x\" (UID: \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:43.052835 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.052787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8712b4-cc01-431f-91cd-c28f0a95b97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x\" (UID: \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:43.053025 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.052881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljsj\" (UniqueName: \"kubernetes.io/projected/6a8712b4-cc01-431f-91cd-c28f0a95b97c-kube-api-access-5ljsj\") pod \"payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x\" (UID: \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:43.053562 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.053544 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8712b4-cc01-431f-91cd-c28f0a95b97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x\" (UID: \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:43.060160 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.060135 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljsj\" (UniqueName: \"kubernetes.io/projected/6a8712b4-cc01-431f-91cd-c28f0a95b97c-kube-api-access-5ljsj\") pod \"payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x\" (UID: \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:43.155523 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.155490 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:43.300127 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.300106 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x"] Jul 28 18:53:43.302778 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:53:43.302752 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8712b4_cc01_431f_91cd_c28f0a95b97c.slice/crio-798282a3f48db27046684da3c7a4121843925f00b0a985e84fab669b74351b52 WatchSource:0}: Error finding container 798282a3f48db27046684da3c7a4121843925f00b0a985e84fab669b74351b52: Status 404 returned error can't find the container with id 798282a3f48db27046684da3c7a4121843925f00b0a985e84fab669b74351b52 Jul 28 18:53:43.571682 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.571595 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" event={"ID":"6a8712b4-cc01-431f-91cd-c28f0a95b97c","Type":"ContainerStarted","Data":"9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f"} Jul 28 18:53:43.571682 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.571678 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" event={"ID":"6a8712b4-cc01-431f-91cd-c28f0a95b97c","Type":"ContainerStarted","Data":"798282a3f48db27046684da3c7a4121843925f00b0a985e84fab669b74351b52"} Jul 28 18:53:43.572158 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.571721 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:53:43.590397 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.590346 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" podStartSLOduration=1.590330936 podStartE2EDuration="1.590330936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:43.587021018 +0000 UTC m=+3168.200305429" watchObservedRunningTime="2026-07-28 18:53:43.590330936 +0000 UTC m=+3168.203615346" Jul 28 18:53:43.976318 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:43.976284 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a6b3d68-71b7-488f-a414-42993e2e3a13" path="/var/lib/kubelet/pods/2a6b3d68-71b7-488f-a414-42993e2e3a13/volumes" Jul 28 18:53:54.579168 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:53:54.579133 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:54:07.858118 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:07.858066 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59"] Jul 28 18:54:07.862303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:07.862276 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" Jul 28 18:54:07.864895 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:07.864872 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:54:07.869848 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:07.869810 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59"] Jul 28 18:54:07.991693 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:07.991653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c8a58-1172-4984-8a6a-37f126ba6dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59\" (UID: \"ad1c8a58-1172-4984-8a6a-37f126ba6dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" Jul 28 18:54:08.093157 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:08.093104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c8a58-1172-4984-8a6a-37f126ba6dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59\" (UID: \"ad1c8a58-1172-4984-8a6a-37f126ba6dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" Jul 28 18:54:08.093788 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:08.093765 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c8a58-1172-4984-8a6a-37f126ba6dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59\" (UID: \"ad1c8a58-1172-4984-8a6a-37f126ba6dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" Jul 28 18:54:08.175628 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:08.175525 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" Jul 28 18:54:08.312675 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:08.312635 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59"] Jul 28 18:54:08.314211 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:54:08.314175 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1c8a58_1172_4984_8a6a_37f126ba6dab.slice/crio-b1d027d0e313aa6852941d8a4059be53a8ce1fe6d80e2ae520222ce2fe31c1d4 WatchSource:0}: Error finding container b1d027d0e313aa6852941d8a4059be53a8ce1fe6d80e2ae520222ce2fe31c1d4: Status 404 returned error can't find the container with id b1d027d0e313aa6852941d8a4059be53a8ce1fe6d80e2ae520222ce2fe31c1d4 Jul 28 18:54:08.722201 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:08.722165 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" event={"ID":"ad1c8a58-1172-4984-8a6a-37f126ba6dab","Type":"ContainerStarted","Data":"17a2413338f59541097c31c8d72efa0e8bf386c6208e50ab20d89fef20996432"} Jul 28 18:54:08.722399 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:08.722223 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" event={"ID":"ad1c8a58-1172-4984-8a6a-37f126ba6dab","Type":"ContainerStarted","Data":"b1d027d0e313aa6852941d8a4059be53a8ce1fe6d80e2ae520222ce2fe31c1d4"} Jul 28 18:54:08.739737 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:08.739682 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" podStartSLOduration=1.739663894 podStartE2EDuration="1.739663894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:54:08.738063315 +0000 UTC m=+3193.351347727" watchObservedRunningTime="2026-07-28 18:54:08.739663894 +0000 UTC m=+3193.352948304" Jul 28 18:54:09.729768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:09.729736 2569 generic.go:356] "Generic (PLEG): container finished" podID="ad1c8a58-1172-4984-8a6a-37f126ba6dab" containerID="17a2413338f59541097c31c8d72efa0e8bf386c6208e50ab20d89fef20996432" exitCode=0 Jul 28 18:54:09.730164 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:09.729828 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" event={"ID":"ad1c8a58-1172-4984-8a6a-37f126ba6dab","Type":"ContainerDied","Data":"17a2413338f59541097c31c8d72efa0e8bf386c6208e50ab20d89fef20996432"} Jul 28 18:54:10.874175 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:10.874148 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" Jul 28 18:54:11.023637 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:11.023546 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c8a58-1172-4984-8a6a-37f126ba6dab-maas-api-service-ca\") pod \"ad1c8a58-1172-4984-8a6a-37f126ba6dab\" (UID: \"ad1c8a58-1172-4984-8a6a-37f126ba6dab\") " Jul 28 18:54:11.023916 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:11.023891 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1c8a58-1172-4984-8a6a-37f126ba6dab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad1c8a58-1172-4984-8a6a-37f126ba6dab" (UID: "ad1c8a58-1172-4984-8a6a-37f126ba6dab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:54:11.124286 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:11.124222 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1c8a58-1172-4984-8a6a-37f126ba6dab-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:54:11.742833 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:11.742799 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" Jul 28 18:54:11.743021 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:11.742834 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59" event={"ID":"ad1c8a58-1172-4984-8a6a-37f126ba6dab","Type":"ContainerDied","Data":"b1d027d0e313aa6852941d8a4059be53a8ce1fe6d80e2ae520222ce2fe31c1d4"} Jul 28 18:54:11.743021 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:11.742889 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1d027d0e313aa6852941d8a4059be53a8ce1fe6d80e2ae520222ce2fe31c1d4" Jul 28 18:54:12.863896 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:12.863863 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x"] Jul 28 18:54:12.864420 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:12.864078 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" podUID="6a8712b4-cc01-431f-91cd-c28f0a95b97c" containerName="payload-pre-processing" containerID="cri-o://9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f" gracePeriod=30 Jul 28 18:54:12.980532 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:12.980500 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59"] Jul 28 18:54:12.983512 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:12.983489 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51b7e8-vkl59"] Jul 28 18:54:13.131017 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.130958 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:54:13.245195 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.245161 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ljsj\" (UniqueName: \"kubernetes.io/projected/6a8712b4-cc01-431f-91cd-c28f0a95b97c-kube-api-access-5ljsj\") pod \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\" (UID: \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\") " Jul 28 18:54:13.245372 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.245215 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8712b4-cc01-431f-91cd-c28f0a95b97c-plugins-config-volume\") pod \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\" (UID: \"6a8712b4-cc01-431f-91cd-c28f0a95b97c\") " Jul 28 18:54:13.245634 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.245612 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8712b4-cc01-431f-91cd-c28f0a95b97c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a8712b4-cc01-431f-91cd-c28f0a95b97c" (UID: "6a8712b4-cc01-431f-91cd-c28f0a95b97c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:54:13.247211 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.247186 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8712b4-cc01-431f-91cd-c28f0a95b97c-kube-api-access-5ljsj" (OuterVolumeSpecName: "kube-api-access-5ljsj") pod "6a8712b4-cc01-431f-91cd-c28f0a95b97c" (UID: "6a8712b4-cc01-431f-91cd-c28f0a95b97c"). InnerVolumeSpecName "kube-api-access-5ljsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:54:13.346332 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.346300 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ljsj\" (UniqueName: \"kubernetes.io/projected/6a8712b4-cc01-431f-91cd-c28f0a95b97c-kube-api-access-5ljsj\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:54:13.346332 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.346323 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8712b4-cc01-431f-91cd-c28f0a95b97c-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:54:13.756309 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.756271 2569 generic.go:356] "Generic (PLEG): container finished" podID="6a8712b4-cc01-431f-91cd-c28f0a95b97c" containerID="9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f" exitCode=0 Jul 28 18:54:13.756309 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.756310 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" Jul 28 18:54:13.756520 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.756358 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" event={"ID":"6a8712b4-cc01-431f-91cd-c28f0a95b97c","Type":"ContainerDied","Data":"9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f"} Jul 28 18:54:13.756520 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.756410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x" event={"ID":"6a8712b4-cc01-431f-91cd-c28f0a95b97c","Type":"ContainerDied","Data":"798282a3f48db27046684da3c7a4121843925f00b0a985e84fab669b74351b52"} Jul 28 18:54:13.756520 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.756433 2569 scope.go:117] "RemoveContainer" containerID="9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f" Jul 28 18:54:13.766023 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.766002 2569 scope.go:117] "RemoveContainer" containerID="9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f" Jul 28 18:54:13.766328 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:54:13.766303 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f\": container with ID starting with 9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f not found: ID does not exist" containerID="9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f" Jul 28 18:54:13.766417 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.766341 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f"} err="failed to get container status \"9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f\": rpc error: code = NotFound desc = could not find container \"9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f\": container with ID starting with 9692da45d1e1d30761ff3c56d6797696e2f09e41f9ae34f110558e6ba929753f not found: ID does not exist" Jul 28 18:54:13.778257 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.778214 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x"] Jul 28 18:54:13.782023 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.781999 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-51b7e8-6fd8d56f76-2p69x"] Jul 28 18:54:13.976414 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.976382 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8712b4-cc01-431f-91cd-c28f0a95b97c" path="/var/lib/kubelet/pods/6a8712b4-cc01-431f-91cd-c28f0a95b97c/volumes" Jul 28 18:54:13.976782 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:13.976713 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad1c8a58-1172-4984-8a6a-37f126ba6dab" path="/var/lib/kubelet/pods/ad1c8a58-1172-4984-8a6a-37f126ba6dab/volumes" Jul 28 18:54:23.227120 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.227089 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94"] Jul 28 18:54:23.227557 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.227542 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8712b4-cc01-431f-91cd-c28f0a95b97c" containerName="payload-pre-processing" Jul 28 18:54:23.227557 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.227558 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8712b4-cc01-431f-91cd-c28f0a95b97c" containerName="payload-pre-processing" Jul 28 18:54:23.227633 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.227572 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1c8a58-1172-4984-8a6a-37f126ba6dab" containerName="revoke-keys" Jul 28 18:54:23.227633 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.227578 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1c8a58-1172-4984-8a6a-37f126ba6dab" containerName="revoke-keys" Jul 28 18:54:23.227700 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.227644 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8712b4-cc01-431f-91cd-c28f0a95b97c" containerName="payload-pre-processing" Jul 28 18:54:23.227700 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.227652 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad1c8a58-1172-4984-8a6a-37f126ba6dab" containerName="revoke-keys" Jul 28 18:54:23.231985 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.231968 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" Jul 28 18:54:23.234734 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.234704 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:54:23.239025 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.238877 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94"] Jul 28 18:54:23.340607 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.340574 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38666820-c284-402d-bb40-e014e1b72c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94\" (UID: \"38666820-c284-402d-bb40-e014e1b72c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" Jul 28 18:54:23.441592 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.441551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38666820-c284-402d-bb40-e014e1b72c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94\" (UID: \"38666820-c284-402d-bb40-e014e1b72c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" Jul 28 18:54:23.442254 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.442214 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38666820-c284-402d-bb40-e014e1b72c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94\" (UID: \"38666820-c284-402d-bb40-e014e1b72c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" Jul 28 18:54:23.544844 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.544752 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" Jul 28 18:54:23.683081 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.683050 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94"] Jul 28 18:54:23.684459 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:54:23.684434 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38666820_c284_402d_bb40_e014e1b72c62.slice/crio-51df4a0c4d51bdb9dab8b76fb4a87caa158ff7266b4e6d127df014272db0ae56 WatchSource:0}: Error finding container 51df4a0c4d51bdb9dab8b76fb4a87caa158ff7266b4e6d127df014272db0ae56: Status 404 returned error can't find the container with id 51df4a0c4d51bdb9dab8b76fb4a87caa158ff7266b4e6d127df014272db0ae56 Jul 28 18:54:23.824877 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:23.824788 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" event={"ID":"38666820-c284-402d-bb40-e014e1b72c62","Type":"ContainerStarted","Data":"51df4a0c4d51bdb9dab8b76fb4a87caa158ff7266b4e6d127df014272db0ae56"} Jul 28 18:54:24.832565 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:24.832481 2569 generic.go:356] "Generic (PLEG): container finished" podID="38666820-c284-402d-bb40-e014e1b72c62" containerID="09b93bce3a2a5aa7b53c1375ad2f178b72fdf421464cfe4330e78260bb4795e3" exitCode=0 Jul 28 18:54:24.832981 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:24.832559 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" event={"ID":"38666820-c284-402d-bb40-e014e1b72c62","Type":"ContainerDied","Data":"09b93bce3a2a5aa7b53c1375ad2f178b72fdf421464cfe4330e78260bb4795e3"} Jul 28 18:54:25.984195 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:25.984171 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" Jul 28 18:54:26.065412 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:26.065381 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38666820-c284-402d-bb40-e014e1b72c62-maas-api-service-ca\") pod \"38666820-c284-402d-bb40-e014e1b72c62\" (UID: \"38666820-c284-402d-bb40-e014e1b72c62\") " Jul 28 18:54:26.065968 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:26.065743 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38666820-c284-402d-bb40-e014e1b72c62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38666820-c284-402d-bb40-e014e1b72c62" (UID: "38666820-c284-402d-bb40-e014e1b72c62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:54:26.166746 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:26.166680 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38666820-c284-402d-bb40-e014e1b72c62-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:54:26.847068 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:26.847026 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" event={"ID":"38666820-c284-402d-bb40-e014e1b72c62","Type":"ContainerDied","Data":"51df4a0c4d51bdb9dab8b76fb4a87caa158ff7266b4e6d127df014272db0ae56"} Jul 28 18:54:26.847068 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:26.847064 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94" Jul 28 18:54:26.847343 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:26.847108 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51df4a0c4d51bdb9dab8b76fb4a87caa158ff7266b4e6d127df014272db0ae56" Jul 28 18:54:38.638328 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.638300 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w"] Jul 28 18:54:38.638820 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.638541 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" podUID="c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" containerName="maas-api" containerID="cri-o://e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3" gracePeriod=30 Jul 28 18:54:38.827397 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.827336 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94"] Jul 28 18:54:38.831675 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.831643 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c1237e-cwr94"] Jul 28 18:54:38.902390 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.902325 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:54:38.922354 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.922317 2569 generic.go:356] "Generic (PLEG): container finished" podID="c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" containerID="e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3" exitCode=0 Jul 28 18:54:38.922568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.922377 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" Jul 28 18:54:38.922568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.922431 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" event={"ID":"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f","Type":"ContainerDied","Data":"e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3"} Jul 28 18:54:38.922568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.922528 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" event={"ID":"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f","Type":"ContainerDied","Data":"cacab888a4528959f1a0277e1740c0e864d7a31eed1e59285c1cd7e7cb5d9eb3"} Jul 28 18:54:38.922568 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.922560 2569 scope.go:117] "RemoveContainer" containerID="e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3" Jul 28 18:54:38.932326 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.932309 2569 scope.go:117] "RemoveContainer" containerID="e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3" Jul 28 18:54:38.932587 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:54:38.932570 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3\": container with ID starting with e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3 not found: ID does not exist" containerID="e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3" Jul 28 18:54:38.932651 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.932600 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3"} err="failed to get container status \"e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3\": rpc error: code = NotFound desc = could not find container \"e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3\": container with ID starting with e7c6ae9c75ec06066b08bb1eb045c3cebe971473202e8bfa38a70e7325b20eb3 not found: ID does not exist" Jul 28 18:54:38.988387 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.988359 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls\") pod \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " Jul 28 18:54:38.988542 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.988423 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-openshift-service-ca\") pod \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " Jul 28 18:54:38.988542 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.988449 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47hrm\" (UniqueName: \"kubernetes.io/projected/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-kube-api-access-47hrm\") pod \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\" (UID: \"c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f\") " Jul 28 18:54:38.988818 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.988792 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" (UID: "c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:54:38.990660 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.990623 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" (UID: "c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:54:38.990769 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:38.990746 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-kube-api-access-47hrm" (OuterVolumeSpecName: "kube-api-access-47hrm") pod "c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" (UID: "c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f"). InnerVolumeSpecName "kube-api-access-47hrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:54:39.089729 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:39.089700 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:54:39.089729 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:39.089726 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:54:39.089918 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:39.089737 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47hrm\" (UniqueName: \"kubernetes.io/projected/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f-kube-api-access-47hrm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:54:39.246683 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:39.246653 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w"] Jul 28 18:54:39.252434 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:39.252408 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w"] Jul 28 18:54:39.976432 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:39.976399 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38666820-c284-402d-bb40-e014e1b72c62" path="/var/lib/kubelet/pods/38666820-c284-402d-bb40-e014e1b72c62/volumes" Jul 28 18:54:39.976819 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:39.976809 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" path="/var/lib/kubelet/pods/c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f/volumes" Jul 28 18:54:41.864919 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:54:41.864860 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c1237e-cb9c56c4c-tmz2w" podUID="c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.63:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 28 18:55:50.837648 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.837611 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx"] Jul 28 18:55:50.838077 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.838050 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" containerName="maas-api" Jul 28 18:55:50.838077 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.838061 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" containerName="maas-api" Jul 28 18:55:50.838167 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.838080 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38666820-c284-402d-bb40-e014e1b72c62" containerName="revoke-keys" Jul 28 18:55:50.838167 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.838086 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38666820-c284-402d-bb40-e014e1b72c62" containerName="revoke-keys" Jul 28 18:55:50.838253 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.838171 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b97d82-08cd-449b-a7e7-3a3f3fc8f88f" containerName="maas-api" Jul 28 18:55:50.838253 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.838178 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="38666820-c284-402d-bb40-e014e1b72c62" containerName="revoke-keys" Jul 28 18:55:50.841367 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.841350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" Jul 28 18:55:50.843976 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.843952 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:55:50.848739 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.848681 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx"] Jul 28 18:55:50.859315 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.859290 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46eba3c7-1696-4e6c-9960-896d21f44b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx\" (UID: \"46eba3c7-1696-4e6c-9960-896d21f44b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" Jul 28 18:55:50.960081 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.960043 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46eba3c7-1696-4e6c-9960-896d21f44b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx\" (UID: \"46eba3c7-1696-4e6c-9960-896d21f44b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" Jul 28 18:55:50.960702 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:50.960683 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46eba3c7-1696-4e6c-9960-896d21f44b59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx\" (UID: \"46eba3c7-1696-4e6c-9960-896d21f44b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" Jul 28 18:55:51.154381 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:51.154351 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" Jul 28 18:55:51.292996 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:51.292959 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx"] Jul 28 18:55:51.294265 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:55:51.294207 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46eba3c7_1696_4e6c_9960_896d21f44b59.slice/crio-ce0b2292ff4d5dd1d6d3c984c2d298bdbdd3e8111c3c425f2a18bc69bae9f569 WatchSource:0}: Error finding container ce0b2292ff4d5dd1d6d3c984c2d298bdbdd3e8111c3c425f2a18bc69bae9f569: Status 404 returned error can't find the container with id ce0b2292ff4d5dd1d6d3c984c2d298bdbdd3e8111c3c425f2a18bc69bae9f569 Jul 28 18:55:51.360707 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:51.360674 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" event={"ID":"46eba3c7-1696-4e6c-9960-896d21f44b59","Type":"ContainerStarted","Data":"ce0b2292ff4d5dd1d6d3c984c2d298bdbdd3e8111c3c425f2a18bc69bae9f569"} Jul 28 18:55:52.374930 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:52.374899 2569 generic.go:356] "Generic (PLEG): container finished" podID="46eba3c7-1696-4e6c-9960-896d21f44b59" containerID="6bf7c94030e0684b058773663d41ddb00b6cb64214f4cc37a0f9b163d84f4718" exitCode=0 Jul 28 18:55:52.375422 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:52.374982 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" event={"ID":"46eba3c7-1696-4e6c-9960-896d21f44b59","Type":"ContainerDied","Data":"6bf7c94030e0684b058773663d41ddb00b6cb64214f4cc37a0f9b163d84f4718"} Jul 28 18:55:53.517873 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:53.517844 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" Jul 28 18:55:53.584517 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:53.584487 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46eba3c7-1696-4e6c-9960-896d21f44b59-maas-api-service-ca\") pod \"46eba3c7-1696-4e6c-9960-896d21f44b59\" (UID: \"46eba3c7-1696-4e6c-9960-896d21f44b59\") " Jul 28 18:55:53.584819 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:53.584797 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46eba3c7-1696-4e6c-9960-896d21f44b59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46eba3c7-1696-4e6c-9960-896d21f44b59" (UID: "46eba3c7-1696-4e6c-9960-896d21f44b59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:55:53.686139 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:53.686046 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46eba3c7-1696-4e6c-9960-896d21f44b59-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:55:54.389139 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:54.389110 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" Jul 28 18:55:54.389321 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:54.389101 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx" event={"ID":"46eba3c7-1696-4e6c-9960-896d21f44b59","Type":"ContainerDied","Data":"ce0b2292ff4d5dd1d6d3c984c2d298bdbdd3e8111c3c425f2a18bc69bae9f569"} Jul 28 18:55:54.389321 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:54.389254 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce0b2292ff4d5dd1d6d3c984c2d298bdbdd3e8111c3c425f2a18bc69bae9f569" Jul 28 18:55:56.205969 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:56.205937 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:55:56.218682 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:55:56.218659 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8lxf9_8b61e46a-8023-4339-b302-2608710d3964/ovn-acl-logging/0.log" Jul 28 18:56:01.252359 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.252327 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf"] Jul 28 18:56:01.252800 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.252576 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" podUID="92d6c15a-38a4-494c-86a3-6229ea6de759" containerName="payload-processing" containerID="cri-o://86223431046c2b9196e717c4a66eaabbc8a6b61c1d615496087422a88a230c43" gracePeriod=30 Jul 28 18:56:01.374803 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.374771 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx"] Jul 28 18:56:01.377192 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.377167 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae953-fdhbx"] Jul 28 18:56:01.434535 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.434501 2569 generic.go:356] "Generic (PLEG): container finished" podID="92d6c15a-38a4-494c-86a3-6229ea6de759" containerID="86223431046c2b9196e717c4a66eaabbc8a6b61c1d615496087422a88a230c43" exitCode=0 Jul 28 18:56:01.434736 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.434541 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" event={"ID":"92d6c15a-38a4-494c-86a3-6229ea6de759","Type":"ContainerDied","Data":"86223431046c2b9196e717c4a66eaabbc8a6b61c1d615496087422a88a230c43"} Jul 28 18:56:01.524393 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.524364 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:56:01.557054 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.557025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d6c15a-38a4-494c-86a3-6229ea6de759-plugins-config-volume\") pod \"92d6c15a-38a4-494c-86a3-6229ea6de759\" (UID: \"92d6c15a-38a4-494c-86a3-6229ea6de759\") " Jul 28 18:56:01.557225 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.557151 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd49n\" (UniqueName: \"kubernetes.io/projected/92d6c15a-38a4-494c-86a3-6229ea6de759-kube-api-access-fd49n\") pod \"92d6c15a-38a4-494c-86a3-6229ea6de759\" (UID: \"92d6c15a-38a4-494c-86a3-6229ea6de759\") " Jul 28 18:56:01.557449 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.557426 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d6c15a-38a4-494c-86a3-6229ea6de759-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92d6c15a-38a4-494c-86a3-6229ea6de759" (UID: "92d6c15a-38a4-494c-86a3-6229ea6de759"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:56:01.559198 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.559178 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d6c15a-38a4-494c-86a3-6229ea6de759-kube-api-access-fd49n" (OuterVolumeSpecName: "kube-api-access-fd49n") pod "92d6c15a-38a4-494c-86a3-6229ea6de759" (UID: "92d6c15a-38a4-494c-86a3-6229ea6de759"). InnerVolumeSpecName "kube-api-access-fd49n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:56:01.657922 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.657888 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd49n\" (UniqueName: \"kubernetes.io/projected/92d6c15a-38a4-494c-86a3-6229ea6de759-kube-api-access-fd49n\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:01.657922 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.657917 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d6c15a-38a4-494c-86a3-6229ea6de759-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:01.976272 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:01.976223 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46eba3c7-1696-4e6c-9960-896d21f44b59" path="/var/lib/kubelet/pods/46eba3c7-1696-4e6c-9960-896d21f44b59/volumes" Jul 28 18:56:02.442512 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:02.442476 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" event={"ID":"92d6c15a-38a4-494c-86a3-6229ea6de759","Type":"ContainerDied","Data":"9dfe2e7d098ccc7cb803efc8933f634c4cdc9d1fa4e8fc1297c25227566f2eaf"} Jul 28 18:56:02.442943 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:02.442533 2569 scope.go:117] "RemoveContainer" containerID="86223431046c2b9196e717c4a66eaabbc8a6b61c1d615496087422a88a230c43" Jul 28 18:56:02.442943 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:02.442491 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf" Jul 28 18:56:02.460352 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:02.460327 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf"] Jul 28 18:56:02.465554 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:02.465532 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae953-78c4dd8fc8-68lcf"] Jul 28 18:56:03.982538 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:03.982502 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d6c15a-38a4-494c-86a3-6229ea6de759" path="/var/lib/kubelet/pods/92d6c15a-38a4-494c-86a3-6229ea6de759/volumes" Jul 28 18:56:07.869130 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:07.869100 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm"] Jul 28 18:56:07.869625 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:07.869376 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" containerID="cri-o://dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4" gracePeriod=30 Jul 28 18:56:12.788542 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:12.788505 2569 patch_prober.go:28] interesting pod/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 18:56:12.788923 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:12.788566 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:56:13.129918 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.129895 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:56:13.274468 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274432 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-certs\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274629 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274497 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-socket\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274629 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274527 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-envoy\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274629 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274552 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b1f347-bc64-404a-ac8a-85256aa0044c-istiod-ca-cert\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274769 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274644 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-credential-socket\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274769 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274671 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-data\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274769 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274696 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-podinfo\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274769 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274731 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl26t\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-kube-api-access-sl26t\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274968 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274772 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:13.274968 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274788 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-token\") pod \"92b1f347-bc64-404a-ac8a-85256aa0044c\" (UID: \"92b1f347-bc64-404a-ac8a-85256aa0044c\") " Jul 28 18:56:13.274968 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.274790 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:13.275135 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.275032 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:13.275192 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.275168 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.275192 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.275187 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-workload-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.275344 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.275201 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-credential-socket\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.275344 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.275217 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b1f347-bc64-404a-ac8a-85256aa0044c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:56:13.275588 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.275565 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-data" (OuterVolumeSpecName: "istio-data") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:13.276867 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.276840 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-kube-api-access-sl26t" (OuterVolumeSpecName: "kube-api-access-sl26t") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "kube-api-access-sl26t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:56:13.276980 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.276887 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:13.277176 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.277157 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-token" (OuterVolumeSpecName: "istio-token") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:56:13.277268 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.277155 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92b1f347-bc64-404a-ac8a-85256aa0044c" (UID: "92b1f347-bc64-404a-ac8a-85256aa0044c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:56:13.375666 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.375639 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-data\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.375666 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.375666 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-podinfo\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.375857 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.375679 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl26t\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-kube-api-access-sl26t\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.375857 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.375692 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-token\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.375857 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.375703 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b1f347-bc64-404a-ac8a-85256aa0044c-istio-envoy\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.375857 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.375713 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b1f347-bc64-404a-ac8a-85256aa0044c-istiod-ca-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:13.508029 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.507992 2569 generic.go:356] "Generic (PLEG): container finished" podID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerID="dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4" exitCode=0 Jul 28 18:56:13.508169 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.508039 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" Jul 28 18:56:13.508169 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.508073 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" event={"ID":"92b1f347-bc64-404a-ac8a-85256aa0044c","Type":"ContainerDied","Data":"dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4"} Jul 28 18:56:13.508169 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.508119 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm" event={"ID":"92b1f347-bc64-404a-ac8a-85256aa0044c","Type":"ContainerDied","Data":"1a867f9899f51480e99f0cf30bfd0123a1158a2e3995fac48373aa550f3a6d0f"} Jul 28 18:56:13.508169 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.508144 2569 scope.go:117] "RemoveContainer" containerID="dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4" Jul 28 18:56:13.517893 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.517877 2569 scope.go:117] "RemoveContainer" containerID="ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4" Jul 28 18:56:13.525936 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.525919 2569 scope.go:117] "RemoveContainer" containerID="dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4" Jul 28 18:56:13.526190 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:56:13.526172 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4\": container with ID starting with dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4 not found: ID does not exist" containerID="dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4" Jul 28 18:56:13.526360 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.526197 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4"} err="failed to get container status \"dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4\": rpc error: code = NotFound desc = could not find container \"dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4\": container with ID starting with dba45ea1c50e0e6842fa497f776f4fc3dd35f5576146f0eb8f436f0c97035fd4 not found: ID does not exist" Jul 28 18:56:13.526360 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.526215 2569 scope.go:117] "RemoveContainer" containerID="ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4" Jul 28 18:56:13.526485 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:56:13.526429 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4\": container with ID starting with ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4 not found: ID does not exist" containerID="ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4" Jul 28 18:56:13.526485 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.526451 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4"} err="failed to get container status \"ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4\": rpc error: code = NotFound desc = could not find container \"ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4\": container with ID starting with ee2e404541cbcf209e4cd89e5a0cb01de3259f5102028c70ce75c78df22072b4 not found: ID does not exist" Jul 28 18:56:13.532644 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.532619 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm"] Jul 28 18:56:13.536261 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.536241 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ae953-openshift-default-6bfbdcdf54-n9vpm"] Jul 28 18:56:13.977488 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:13.977445 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" path="/var/lib/kubelet/pods/92b1f347-bc64-404a-ac8a-85256aa0044c/volumes" Jul 28 18:56:18.745087 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:18.744998 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns"] Jul 28 18:56:18.747311 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:18.747288 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-37e11d-29754405-vwhns"] Jul 28 18:56:19.976091 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:19.976055 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="109d4a5b-527a-409b-94b4-8318705764da" path="/var/lib/kubelet/pods/109d4a5b-527a-409b-94b4-8318705764da/volumes" Jul 28 18:56:25.868573 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.868539 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc"] Jul 28 18:56:25.869063 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.868955 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" Jul 28 18:56:25.869063 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.868966 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" Jul 28 18:56:25.869063 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.868974 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46eba3c7-1696-4e6c-9960-896d21f44b59" containerName="revoke-keys" Jul 28 18:56:25.869063 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.868980 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="46eba3c7-1696-4e6c-9960-896d21f44b59" containerName="revoke-keys" Jul 28 18:56:25.869063 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.868996 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d6c15a-38a4-494c-86a3-6229ea6de759" containerName="payload-processing" Jul 28 18:56:25.869063 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.869002 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d6c15a-38a4-494c-86a3-6229ea6de759" containerName="payload-processing" Jul 28 18:56:25.869282 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.869069 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" Jul 28 18:56:25.869282 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.869078 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="46eba3c7-1696-4e6c-9960-896d21f44b59" containerName="revoke-keys" Jul 28 18:56:25.869282 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.869084 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d6c15a-38a4-494c-86a3-6229ea6de759" containerName="payload-processing" Jul 28 18:56:25.873653 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.873635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" Jul 28 18:56:25.876900 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.876877 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvg8j\"" Jul 28 18:56:25.880385 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.880361 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc"] Jul 28 18:56:25.992471 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:25.992443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6579253-1d48-4c3e-af0b-d781dc0db5ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc\" (UID: \"d6579253-1d48-4c3e-af0b-d781dc0db5ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" Jul 28 18:56:26.093447 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:26.093409 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6579253-1d48-4c3e-af0b-d781dc0db5ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc\" (UID: \"d6579253-1d48-4c3e-af0b-d781dc0db5ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" Jul 28 18:56:26.094046 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:26.094021 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6579253-1d48-4c3e-af0b-d781dc0db5ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc\" (UID: \"d6579253-1d48-4c3e-af0b-d781dc0db5ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" Jul 28 18:56:26.186303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:26.186189 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" Jul 28 18:56:26.324155 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:26.324126 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc"] Jul 28 18:56:26.325812 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:56:26.325787 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6579253_1d48_4c3e_af0b_d781dc0db5ed.slice/crio-4f2843f3406e0f1a040fc2db6f13f3483daa986e9bc24c972d084f79ab2b6461 WatchSource:0}: Error finding container 4f2843f3406e0f1a040fc2db6f13f3483daa986e9bc24c972d084f79ab2b6461: Status 404 returned error can't find the container with id 4f2843f3406e0f1a040fc2db6f13f3483daa986e9bc24c972d084f79ab2b6461 Jul 28 18:56:26.584192 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:26.584159 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" event={"ID":"d6579253-1d48-4c3e-af0b-d781dc0db5ed","Type":"ContainerStarted","Data":"4f2843f3406e0f1a040fc2db6f13f3483daa986e9bc24c972d084f79ab2b6461"} Jul 28 18:56:27.591756 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:27.591718 2569 generic.go:356] "Generic (PLEG): container finished" podID="d6579253-1d48-4c3e-af0b-d781dc0db5ed" containerID="6ab3545017eb0755aa0782b4ad7d0eb35037a2d763bac751cb033366a27c21f4" exitCode=0 Jul 28 18:56:27.592153 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:27.591810 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" event={"ID":"d6579253-1d48-4c3e-af0b-d781dc0db5ed","Type":"ContainerDied","Data":"6ab3545017eb0755aa0782b4ad7d0eb35037a2d763bac751cb033366a27c21f4"} Jul 28 18:56:28.740657 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:28.740623 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" Jul 28 18:56:28.920473 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:28.920385 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6579253-1d48-4c3e-af0b-d781dc0db5ed-maas-api-service-ca\") pod \"d6579253-1d48-4c3e-af0b-d781dc0db5ed\" (UID: \"d6579253-1d48-4c3e-af0b-d781dc0db5ed\") " Jul 28 18:56:28.920706 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:28.920682 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6579253-1d48-4c3e-af0b-d781dc0db5ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6579253-1d48-4c3e-af0b-d781dc0db5ed" (UID: "d6579253-1d48-4c3e-af0b-d781dc0db5ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:56:29.021867 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:29.021836 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6579253-1d48-4c3e-af0b-d781dc0db5ed-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:29.605196 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:29.605162 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" event={"ID":"d6579253-1d48-4c3e-af0b-d781dc0db5ed","Type":"ContainerDied","Data":"4f2843f3406e0f1a040fc2db6f13f3483daa986e9bc24c972d084f79ab2b6461"} Jul 28 18:56:29.605196 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:29.605187 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc" Jul 28 18:56:29.605433 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:29.605213 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f2843f3406e0f1a040fc2db6f13f3483daa986e9bc24c972d084f79ab2b6461" Jul 28 18:56:36.481736 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.481694 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7"] Jul 28 18:56:36.485511 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.485478 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f36101-29754405-226l7"] Jul 28 18:56:36.511802 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.511765 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx"] Jul 28 18:56:36.632704 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.632669 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc"] Jul 28 18:56:36.635019 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.634994 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f36101-tgwkc"] Jul 28 18:56:36.645818 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.645781 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" podUID="e48a897c-f4cb-4362-b4f9-460d2dca4128" containerName="payload-pre-processing" containerID="cri-o://8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46" gracePeriod=30 Jul 28 18:56:36.801289 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.801250 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" podUID="e48a897c-f4cb-4362-b4f9-460d2dca4128" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.59:9004: connect: connection refused" Jul 28 18:56:36.924327 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.924301 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:56:36.999076 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.999027 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr6z2\" (UniqueName: \"kubernetes.io/projected/e48a897c-f4cb-4362-b4f9-460d2dca4128-kube-api-access-kr6z2\") pod \"e48a897c-f4cb-4362-b4f9-460d2dca4128\" (UID: \"e48a897c-f4cb-4362-b4f9-460d2dca4128\") " Jul 28 18:56:36.999289 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.999176 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48a897c-f4cb-4362-b4f9-460d2dca4128-plugins-config-volume\") pod \"e48a897c-f4cb-4362-b4f9-460d2dca4128\" (UID: \"e48a897c-f4cb-4362-b4f9-460d2dca4128\") " Jul 28 18:56:36.999617 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.999587 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48a897c-f4cb-4362-b4f9-460d2dca4128-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e48a897c-f4cb-4362-b4f9-460d2dca4128" (UID: "e48a897c-f4cb-4362-b4f9-460d2dca4128"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:56:36.999874 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:36.999852 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48a897c-f4cb-4362-b4f9-460d2dca4128-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:37.001335 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.001303 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48a897c-f4cb-4362-b4f9-460d2dca4128-kube-api-access-kr6z2" (OuterVolumeSpecName: "kube-api-access-kr6z2") pod "e48a897c-f4cb-4362-b4f9-460d2dca4128" (UID: "e48a897c-f4cb-4362-b4f9-460d2dca4128"). InnerVolumeSpecName "kube-api-access-kr6z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:56:37.101067 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.101033 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6z2\" (UniqueName: \"kubernetes.io/projected/e48a897c-f4cb-4362-b4f9-460d2dca4128-kube-api-access-kr6z2\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Jul 28 18:56:37.653141 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.653103 2569 generic.go:356] "Generic (PLEG): container finished" podID="e48a897c-f4cb-4362-b4f9-460d2dca4128" containerID="8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46" exitCode=0 Jul 28 18:56:37.653579 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.653153 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" Jul 28 18:56:37.653579 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.653183 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" event={"ID":"e48a897c-f4cb-4362-b4f9-460d2dca4128","Type":"ContainerDied","Data":"8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46"} Jul 28 18:56:37.653579 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.653254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx" event={"ID":"e48a897c-f4cb-4362-b4f9-460d2dca4128","Type":"ContainerDied","Data":"687fae681c3ceef24a71f0f6ff1b61450618c9463dc883b65024a3e0fd3420e8"} Jul 28 18:56:37.653579 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.653272 2569 scope.go:117] "RemoveContainer" containerID="8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46" Jul 28 18:56:37.663046 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.663029 2569 scope.go:117] "RemoveContainer" containerID="8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46" Jul 28 18:56:37.663327 ip-10-0-139-201 kubenswrapper[2569]: E0728 18:56:37.663305 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46\": container with ID starting with 8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46 not found: ID does not exist" containerID="8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46" Jul 28 18:56:37.663398 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.663336 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46"} err="failed to get container status \"8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46\": rpc error: code = NotFound desc = could not find container \"8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46\": container with ID starting with 8d3b1df722544d90c879b1327d5171b8c13f7415694f001a84446fb858ff9e46 not found: ID does not exist" Jul 28 18:56:37.675897 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.675871 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx"] Jul 28 18:56:37.679468 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.679445 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-f36101-7f6c697f44-j95lx"] Jul 28 18:56:37.975947 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.975847 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18131847-6944-4c99-b81c-3c0fa8e6434c" path="/var/lib/kubelet/pods/18131847-6944-4c99-b81c-3c0fa8e6434c/volumes" Jul 28 18:56:37.976281 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.976261 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6579253-1d48-4c3e-af0b-d781dc0db5ed" path="/var/lib/kubelet/pods/d6579253-1d48-4c3e-af0b-d781dc0db5ed/volumes" Jul 28 18:56:37.976648 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:37.976631 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48a897c-f4cb-4362-b4f9-460d2dca4128" path="/var/lib/kubelet/pods/e48a897c-f4cb-4362-b4f9-460d2dca4128/volumes" Jul 28 18:56:49.252011 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:49.251982 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xwkbh_fc15845f-7f33-4334-8531-6da0357fd1b3/manager/0.log" Jul 28 18:56:49.367447 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:49.367417 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mnn5s_5505cefc-5b40-4812-be42-aa884e5e176c/manager/0.log" Jul 28 18:56:49.613363 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:49.613335 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vq8ws_7192e8d6-283a-4eae-b732-982758f8c802/server/0.log" Jul 28 18:56:49.730310 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:49.730281 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kft7m_43d52037-1291-4aa3-977d-dfa65ca62b05/manager/0.log" Jul 28 18:56:49.844637 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:49.844610 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7795fd7c74-nt97l_235a57a7-ce35-46fb-95b7-1d4331b2c1e0/copy-manifests/0.log" Jul 28 18:56:49.850660 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:49.850639 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7795fd7c74-nt97l_235a57a7-ce35-46fb-95b7-1d4331b2c1e0/manager/0.log" Jul 28 18:56:50.423019 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:50.422988 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6fd64c844b-dfmn8_43ad43d8-f609-4421-a3f7-15b27bc6cd51/maas-api/0.log" Jul 28 18:56:50.755645 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:50.755569 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754405-qxz9m_903afd21-dbb2-4e0f-85d1-59fb138a34b8/cleanup/0.log" Jul 28 18:56:50.866021 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:50.865991 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wlh9d_46f63170-4224-4610-bcaa-fc9c9047c27d/postgres/0.log" Jul 28 18:56:51.656990 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.656963 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7_0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41/pull/0.log" Jul 28 18:56:51.662264 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.662245 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7_0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41/extract/0.log" Jul 28 18:56:51.667013 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.666993 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7_0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41/util/0.log" Jul 28 18:56:51.778102 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.778080 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v_5cc0dbbc-8350-4d82-b3a6-a02ea061b488/util/0.log" Jul 28 18:56:51.783398 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.783378 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v_5cc0dbbc-8350-4d82-b3a6-a02ea061b488/pull/0.log" Jul 28 18:56:51.789171 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.789156 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v_5cc0dbbc-8350-4d82-b3a6-a02ea061b488/extract/0.log" Jul 28 18:56:51.904108 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.904085 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t_967cfe7b-b368-4594-94e2-47d888747e32/util/0.log" Jul 28 18:56:51.909601 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.909511 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t_967cfe7b-b368-4594-94e2-47d888747e32/pull/0.log" Jul 28 18:56:51.914336 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:51.914317 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t_967cfe7b-b368-4594-94e2-47d888747e32/extract/0.log" Jul 28 18:56:52.023241 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:52.023202 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw_71d4f93c-8915-4762-89a8-761f19435806/util/0.log" Jul 28 18:56:52.028797 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:52.028770 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw_71d4f93c-8915-4762-89a8-761f19435806/pull/0.log" Jul 28 18:56:52.034093 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:52.034073 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw_71d4f93c-8915-4762-89a8-761f19435806/extract/0.log" Jul 28 18:56:52.959988 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:52.959955 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tlgkx_7cc8d485-ec21-41a2-a692-bd9e04394018/manager/0.log" Jul 28 18:56:53.415245 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:53.415175 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-954rj_c6616f83-1226-42c4-8dc1-aa7b02b15589/discovery/0.log" Jul 28 18:56:53.529069 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:53.529041 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b97f775b-kg9ck_4ac59740-707e-4a48-8dec-ad3a21a89be0/kube-auth-proxy/0.log" Jul 28 18:56:53.875035 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:53.875007 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-r2txr_9eb1a8e1-f6ff-4ca0-8634-1492f212cad2/payload-pre-processing/0.log" Jul 28 18:56:54.005665 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:54.005637 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9kkbx_b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59/payload-processing/0.log" Jul 28 18:56:54.693330 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:54.693302 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-hz9hb_8f6cd429-3f1b-480a-a40f-7cdedd3ebf02/main/0.log" Jul 28 18:56:54.908174 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:54.908145 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lglmq_6cb10d46-8b88-4f53-82e4-4925dcd6b9a0/main/0.log" Jul 28 18:56:56.399945 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:56.399918 2569 scope.go:117] "RemoveContainer" containerID="dc979abc7dcc91e5e9b717d81508b5b5a96dde6e62f9e132d7d082bd9e89fe69" Jul 28 18:56:56.408894 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:56:56.408874 2569 scope.go:117] "RemoveContainer" containerID="67ec247db14569051a5724847806272def68e80cc2af42bd19aa64d1966b5481" Jul 28 18:57:03.063165 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:03.063132 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jw4r5_8fd18dc0-806e-46c1-811c-31680ca4fbdb/global-pull-secret-syncer/0.log" Jul 28 18:57:03.311238 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:03.311197 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vn5cx_62704919-bf62-4605-b9ee-8749f46bb72d/konnectivity-agent/0.log" Jul 28 18:57:03.330794 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:03.330724 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-201.ec2.internal_f02759f6c62b6d84c775f3fa1d163393/haproxy/0.log" Jul 28 18:57:07.700493 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.700462 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7_0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41/extract/0.log" Jul 28 18:57:07.721366 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.721339 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7_0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41/util/0.log" Jul 28 18:57:07.744983 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.744961 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hkjx7_0f9fea1b-bf7a-4e8b-9f27-5c23abea8f41/pull/0.log" Jul 28 18:57:07.771580 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.771560 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v_5cc0dbbc-8350-4d82-b3a6-a02ea061b488/extract/0.log" Jul 28 18:57:07.796737 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.796710 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v_5cc0dbbc-8350-4d82-b3a6-a02ea061b488/util/0.log" Jul 28 18:57:07.819844 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.819820 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e042j9v_5cc0dbbc-8350-4d82-b3a6-a02ea061b488/pull/0.log" Jul 28 18:57:07.853240 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.853214 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t_967cfe7b-b368-4594-94e2-47d888747e32/extract/0.log" Jul 28 18:57:07.874289 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.874263 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t_967cfe7b-b368-4594-94e2-47d888747e32/util/0.log" Jul 28 18:57:07.892640 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.892604 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wz77t_967cfe7b-b368-4594-94e2-47d888747e32/pull/0.log" Jul 28 18:57:07.920795 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.920777 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw_71d4f93c-8915-4762-89a8-761f19435806/extract/0.log" Jul 28 18:57:07.939469 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.939447 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw_71d4f93c-8915-4762-89a8-761f19435806/util/0.log" Jul 28 18:57:07.958118 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:07.958032 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1wthcw_71d4f93c-8915-4762-89a8-761f19435806/pull/0.log" Jul 28 18:57:08.989575 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:08.989543 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tlgkx_7cc8d485-ec21-41a2-a692-bd9e04394018/manager/0.log" Jul 28 18:57:10.928731 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:10.928702 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wskhq_27abc741-63d6-450e-a598-84ac1d9a990c/kube-state-metrics/0.log" Jul 28 18:57:10.945114 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:10.945092 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wskhq_27abc741-63d6-450e-a598-84ac1d9a990c/kube-rbac-proxy-main/0.log" Jul 28 18:57:10.963903 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:10.963884 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wskhq_27abc741-63d6-450e-a598-84ac1d9a990c/kube-rbac-proxy-self/0.log" Jul 28 18:57:11.201023 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.200941 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkjtz_c1d0895b-9fc5-44a0-9337-53f0f3aaf32f/node-exporter/0.log" Jul 28 18:57:11.219040 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.219019 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkjtz_c1d0895b-9fc5-44a0-9337-53f0f3aaf32f/kube-rbac-proxy/0.log" Jul 28 18:57:11.239357 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.239335 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkjtz_c1d0895b-9fc5-44a0-9337-53f0f3aaf32f/init-textfile/0.log" Jul 28 18:57:11.264609 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.264592 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jlrct_ccde900f-54ed-4219-8aca-c86afdc74895/kube-rbac-proxy-main/0.log" Jul 28 18:57:11.288983 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.288961 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jlrct_ccde900f-54ed-4219-8aca-c86afdc74895/kube-rbac-proxy-self/0.log" Jul 28 18:57:11.307680 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.307660 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jlrct_ccde900f-54ed-4219-8aca-c86afdc74895/openshift-state-metrics/0.log" Jul 28 18:57:11.547956 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.547877 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bd4b8bff-7vqc6_f89be472-d3aa-44a5-8d22-068bdea37f86/telemeter-client/0.log" Jul 28 18:57:11.567565 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.567542 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bd4b8bff-7vqc6_f89be472-d3aa-44a5-8d22-068bdea37f86/reload/0.log" Jul 28 18:57:11.586303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.586281 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bd4b8bff-7vqc6_f89be472-d3aa-44a5-8d22-068bdea37f86/kube-rbac-proxy/0.log" Jul 28 18:57:11.613028 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.613005 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f694f7dcd-gqdrn_f425edc9-5a21-4128-8067-0563c2c83e1e/thanos-query/0.log" Jul 28 18:57:11.631162 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.631140 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f694f7dcd-gqdrn_f425edc9-5a21-4128-8067-0563c2c83e1e/kube-rbac-proxy-web/0.log" Jul 28 18:57:11.651792 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.651766 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f694f7dcd-gqdrn_f425edc9-5a21-4128-8067-0563c2c83e1e/kube-rbac-proxy/0.log" Jul 28 18:57:11.669984 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.669960 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f694f7dcd-gqdrn_f425edc9-5a21-4128-8067-0563c2c83e1e/prom-label-proxy/0.log" Jul 28 18:57:11.688103 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.688081 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f694f7dcd-gqdrn_f425edc9-5a21-4128-8067-0563c2c83e1e/kube-rbac-proxy-rules/0.log" Jul 28 18:57:11.706951 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:11.706929 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f694f7dcd-gqdrn_f425edc9-5a21-4128-8067-0563c2c83e1e/kube-rbac-proxy-metrics/0.log" Jul 28 18:57:12.824282 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:12.824255 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-62scc_ab1429a6-af73-45b1-acff-7c8c53553881/networking-console-plugin/0.log" Jul 28 18:57:13.075133 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075051 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9"] Jul 28 18:57:13.075529 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075514 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48a897c-f4cb-4362-b4f9-460d2dca4128" containerName="payload-pre-processing" Jul 28 18:57:13.075587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075530 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48a897c-f4cb-4362-b4f9-460d2dca4128" containerName="payload-pre-processing" Jul 28 18:57:13.075587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075557 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6579253-1d48-4c3e-af0b-d781dc0db5ed" containerName="revoke-keys" Jul 28 18:57:13.075587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075562 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6579253-1d48-4c3e-af0b-d781dc0db5ed" containerName="revoke-keys" Jul 28 18:57:13.075587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075568 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" Jul 28 18:57:13.075587 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075574 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" Jul 28 18:57:13.075768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075639 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b1f347-bc64-404a-ac8a-85256aa0044c" containerName="istio-proxy" Jul 28 18:57:13.075768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075649 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48a897c-f4cb-4362-b4f9-460d2dca4128" containerName="payload-pre-processing" Jul 28 18:57:13.075768 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.075657 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6579253-1d48-4c3e-af0b-d781dc0db5ed" containerName="revoke-keys" Jul 28 18:57:13.078822 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.078807 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.081450 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.081428 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-22fmk\"/\"openshift-service-ca.crt\"" Jul 28 18:57:13.081597 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.081454 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-22fmk\"/\"kube-root-ca.crt\"" Jul 28 18:57:13.082811 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.082790 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-22fmk\"/\"default-dockercfg-fc5m6\"" Jul 28 18:57:13.085460 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.085438 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9"] Jul 28 18:57:13.225126 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.225089 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-podres\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.225303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.225141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-lib-modules\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.225303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.225169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-proc\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.225303 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.225283 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-sys\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.225448 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.225349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nx8h\" (UniqueName: \"kubernetes.io/projected/4fe82151-a1ca-4340-a31b-e1ffbb961c25-kube-api-access-9nx8h\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326416 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-podres\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326416 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-lib-modules\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326704 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326447 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-proc\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326704 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326514 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-sys\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326704 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326524 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-podres\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326704 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nx8h\" (UniqueName: \"kubernetes.io/projected/4fe82151-a1ca-4340-a31b-e1ffbb961c25-kube-api-access-9nx8h\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326704 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326589 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-proc\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326704 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326669 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-sys\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.326920 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.326720 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fe82151-a1ca-4340-a31b-e1ffbb961c25-lib-modules\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.334547 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.334521 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nx8h\" (UniqueName: \"kubernetes.io/projected/4fe82151-a1ca-4340-a31b-e1ffbb961c25-kube-api-access-9nx8h\") pod \"perf-node-gather-daemonset-5swd9\" (UID: \"4fe82151-a1ca-4340-a31b-e1ffbb961c25\") " pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.390394 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.390368 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.525452 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.525425 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9"] Jul 28 18:57:13.526707 ip-10-0-139-201 kubenswrapper[2569]: W0728 18:57:13.526678 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4fe82151_a1ca_4340_a31b_e1ffbb961c25.slice/crio-bb16715e950a497ce4de666bcaad05d14eb901560ad2411f8308c0fb6fc8c50c WatchSource:0}: Error finding container bb16715e950a497ce4de666bcaad05d14eb901560ad2411f8308c0fb6fc8c50c: Status 404 returned error can't find the container with id bb16715e950a497ce4de666bcaad05d14eb901560ad2411f8308c0fb6fc8c50c Jul 28 18:57:13.870736 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.870704 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" event={"ID":"4fe82151-a1ca-4340-a31b-e1ffbb961c25","Type":"ContainerStarted","Data":"624a4690f233d654c57f0b85c6358af9afb8761a5390c629afbfc8dd3ab871d5"} Jul 28 18:57:13.871104 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.870756 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" event={"ID":"4fe82151-a1ca-4340-a31b-e1ffbb961c25","Type":"ContainerStarted","Data":"bb16715e950a497ce4de666bcaad05d14eb901560ad2411f8308c0fb6fc8c50c"} Jul 28 18:57:13.871104 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.870865 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:13.888376 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:13.888329 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" podStartSLOduration=0.88830829 podStartE2EDuration="888.30829ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:57:13.885369815 +0000 UTC m=+3378.498654224" watchObservedRunningTime="2026-07-28 18:57:13.88830829 +0000 UTC m=+3378.501592700" Jul 28 18:57:14.029456 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:14.029374 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-c9579_aa052ba4-aa2e-42ff-9ec8-ce2472d04d2e/download-server/0.log" Jul 28 18:57:15.604064 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:15.604039 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdbm7_7e4537df-f2f4-438f-9a20-94d0d9728285/dns/0.log" Jul 28 18:57:15.623646 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:15.623624 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdbm7_7e4537df-f2f4-438f-9a20-94d0d9728285/kube-rbac-proxy/0.log" Jul 28 18:57:15.735709 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:15.735675 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w829b_d42d3962-0660-48ee-8248-66d18ead7206/dns-node-resolver/0.log" Jul 28 18:57:16.238954 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:16.238920 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8675b75794-2vvzv_13ad361a-b8bf-4962-9436-bdfa0a101eb8/registry/0.log" Jul 28 18:57:16.260111 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:16.260085 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ds5hw_bb20a3ca-9707-403d-a6d1-6093065b2a46/node-ca/0.log" Jul 28 18:57:17.351163 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:17.351129 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-954rj_c6616f83-1226-42c4-8dc1-aa7b02b15589/discovery/0.log" Jul 28 18:57:17.370783 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:17.370757 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b97f775b-kg9ck_4ac59740-707e-4a48-8dec-ad3a21a89be0/kube-auth-proxy/0.log" Jul 28 18:57:17.515980 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:17.515950 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-r2txr_9eb1a8e1-f6ff-4ca0-8634-1492f212cad2/payload-pre-processing/0.log" Jul 28 18:57:17.545100 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:17.545073 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9kkbx_b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59/payload-processing/0.log" Jul 28 18:57:18.328735 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:18.328707 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9h8xs_a7f70237-bb77-49a6-8943-8446de31eb9e/serve-healthcheck-canary/0.log" Jul 28 18:57:18.902344 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:18.902315 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx5c9_27917fe9-74f7-487d-863c-9725cf62b6a4/kube-rbac-proxy/0.log" Jul 28 18:57:18.921658 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:18.921635 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx5c9_27917fe9-74f7-487d-863c-9725cf62b6a4/exporter/0.log" Jul 28 18:57:18.941128 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:18.941102 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx5c9_27917fe9-74f7-487d-863c-9725cf62b6a4/extractor/0.log" Jul 28 18:57:19.886791 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:19.886764 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-22fmk/perf-node-gather-daemonset-5swd9" Jul 28 18:57:21.257642 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:21.257612 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xwkbh_fc15845f-7f33-4334-8531-6da0357fd1b3/manager/0.log" Jul 28 18:57:21.473995 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:21.473962 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mnn5s_5505cefc-5b40-4812-be42-aa884e5e176c/manager/0.log" Jul 28 18:57:21.768951 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:21.768923 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vq8ws_7192e8d6-283a-4eae-b732-982758f8c802/server/0.log" Jul 28 18:57:21.834765 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:21.834732 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kft7m_43d52037-1291-4aa3-977d-dfa65ca62b05/manager/0.log" Jul 28 18:57:21.856134 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:21.856110 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7795fd7c74-nt97l_235a57a7-ce35-46fb-95b7-1d4331b2c1e0/manager/0.log" Jul 28 18:57:21.875396 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:21.875376 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7795fd7c74-nt97l_235a57a7-ce35-46fb-95b7-1d4331b2c1e0/copy-manifests/0.log" Jul 28 18:57:23.471466 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:23.471433 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-954rj_c6616f83-1226-42c4-8dc1-aa7b02b15589/discovery/0.log" Jul 28 18:57:23.481493 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:23.481471 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b97f775b-kg9ck_4ac59740-707e-4a48-8dec-ad3a21a89be0/kube-auth-proxy/0.log" Jul 28 18:57:23.589007 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:23.588983 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-r2txr_9eb1a8e1-f6ff-4ca0-8634-1492f212cad2/payload-pre-processing/0.log" Jul 28 18:57:23.605940 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:23.605915 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9kkbx_b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59/payload-processing/0.log" Jul 28 18:57:25.076955 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:25.076869 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-954rj_c6616f83-1226-42c4-8dc1-aa7b02b15589/discovery/0.log" Jul 28 18:57:25.089489 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:25.089470 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b97f775b-kg9ck_4ac59740-707e-4a48-8dec-ad3a21a89be0/kube-auth-proxy/0.log" Jul 28 18:57:25.201905 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:25.201874 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-r2txr_9eb1a8e1-f6ff-4ca0-8634-1492f212cad2/payload-pre-processing/0.log" Jul 28 18:57:25.220041 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:25.220015 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9kkbx_b9f8aa4e-8c2b-478d-8c14-6d8d6518fc59/payload-processing/0.log" Jul 28 18:57:28.671141 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:28.671086 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq_ec8ee9bc-7ba6-482d-9a6e-4a944173680e/kube-storage-version-migrator-operator/1.log" Jul 28 18:57:28.672135 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:28.672116 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-hftmq_ec8ee9bc-7ba6-482d-9a6e-4a944173680e/kube-storage-version-migrator-operator/0.log" Jul 28 18:57:29.856989 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:29.856959 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84xx7_43195a1e-92f0-4c69-bef3-c529d4c75fd8/kube-multus-additional-cni-plugins/0.log" Jul 28 18:57:29.878857 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:29.878829 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84xx7_43195a1e-92f0-4c69-bef3-c529d4c75fd8/egress-router-binary-copy/0.log" Jul 28 18:57:29.901289 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:29.901265 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84xx7_43195a1e-92f0-4c69-bef3-c529d4c75fd8/cni-plugins/0.log" Jul 28 18:57:29.921565 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:29.921540 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84xx7_43195a1e-92f0-4c69-bef3-c529d4c75fd8/bond-cni-plugin/0.log" Jul 28 18:57:29.943936 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:29.943913 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84xx7_43195a1e-92f0-4c69-bef3-c529d4c75fd8/routeoverride-cni/0.log" Jul 28 18:57:29.967590 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:29.967565 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84xx7_43195a1e-92f0-4c69-bef3-c529d4c75fd8/whereabouts-cni-bincopy/0.log" Jul 28 18:57:29.988649 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:29.988628 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-84xx7_43195a1e-92f0-4c69-bef3-c529d4c75fd8/whereabouts-cni/0.log" Jul 28 18:57:30.206836 ip-10-0-139-201 kubenswrapper[2569]: I0728 18:57:30.206735 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pvms7_057dfc76-9edb-4849-9f5c-f33869d37b32/kube-multus/0.log"