Aug 01 00:33:40.627239 ip-10-0-137-42 systemd[1]: Starting Kubernetes Kubelet... Aug 01 00:33:40.982690 ip-10-0-137-42 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:33:40.982690 ip-10-0-137-42 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 01 00:33:40.982690 ip-10-0-137-42 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:33:40.982690 ip-10-0-137-42 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 01 00:33:40.982690 ip-10-0-137-42 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:33:40.983738 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.983648 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 01 00:33:40.990319 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990292 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:33:40.990319 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990314 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:33:40.990319 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990319 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:33:40.990319 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990323 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:33:40.990319 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990327 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990330 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990333 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990337 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990341 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990344 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990347 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990350 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990353 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990355 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990358 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990360 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990363 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990365 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990368 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990370 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990373 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990375 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990378 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:33:40.990534 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990380 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990383 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990385 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990388 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990390 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990393 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990395 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990398 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990405 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990408 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990410 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990413 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990416 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990419 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990422 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990424 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990427 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990429 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990432 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990434 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:33:40.991005 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990437 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990439 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990442 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990444 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990446 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990449 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990451 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990454 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990456 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990459 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990462 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990464 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990466 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990469 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990472 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990476 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990480 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990482 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990485 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:33:40.991522 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990487 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990489 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990492 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990494 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990497 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990501 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990504 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990506 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990510 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990513 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990515 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990518 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990520 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990523 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990525 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990527 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990530 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990532 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990535 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990537 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:33:40.991993 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990540 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990544 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990546 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990549 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990551 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990554 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990556 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990559 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990563 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990565 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990568 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990570 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990573 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990575 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990577 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990580 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.990582 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992141 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992149 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992153 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:33:40.992477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992156 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992159 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992162 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992164 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992167 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992169 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992172 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992174 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992177 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992179 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992182 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992185 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992187 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992189 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992195 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992198 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992200 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992203 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992205 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992208 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:33:40.993038 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992210 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992213 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992216 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992218 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992221 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992223 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992227 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992230 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992233 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992235 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992238 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992243 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992247 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992250 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992253 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992255 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992258 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992261 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992263 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:33:40.993593 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992266 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992268 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992271 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992273 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992275 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992278 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992281 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992284 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992288 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992291 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992294 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992296 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992299 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992301 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992304 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992306 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992309 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992311 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992314 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992317 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:33:40.994069 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992319 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992322 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992324 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992327 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992329 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992331 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992334 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992337 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992339 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992341 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992345 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992348 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992350 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992352 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992355 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992357 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992360 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992362 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992364 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992368 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:33:40.994548 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992371 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992374 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992376 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992379 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992382 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992385 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992388 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992390 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992394 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992397 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992400 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992403 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992405 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992408 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992411 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992413 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992416 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992811 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992816 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:33:40.995060 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992821 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992824 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992827 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992831 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992833 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992836 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992839 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992842 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992844 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992847 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992849 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992852 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992856 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992859 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992862 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992864 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992867 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992869 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992872 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:33:40.995519 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992874 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992877 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992881 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992885 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992887 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992890 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992892 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992895 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992897 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992900 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992902 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992905 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992923 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992926 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992929 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992931 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992934 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992936 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992939 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:33:40.995998 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992941 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992944 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992946 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992949 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992951 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992954 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992957 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992960 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992962 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992965 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992968 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992970 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992973 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992975 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992978 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992981 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992983 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992989 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992991 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992994 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:33:40.996471 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992996 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.992999 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993001 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993003 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993006 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993008 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993011 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993013 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993015 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993018 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993021 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993025 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993028 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993031 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993034 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993036 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993039 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993042 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993046 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993049 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:33:40.996952 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993052 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993054 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993058 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993060 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993063 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993065 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993068 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993070 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993073 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993075 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993077 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993080 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993082 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993085 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993087 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993090 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993092 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993094 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993097 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993170 2568 flags.go:64] FLAG: --address="0.0.0.0" Aug 01 00:33:40.997431 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993177 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993184 2568 flags.go:64] FLAG: --anonymous-auth="true" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993191 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993198 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993203 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993209 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993216 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993221 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993224 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993227 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993231 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993234 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993237 2568 flags.go:64] FLAG: --cgroup-root="" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993239 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993242 2568 flags.go:64] FLAG: --client-ca-file="" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993250 2568 flags.go:64] FLAG: --cloud-provider="external" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993253 2568 flags.go:64] FLAG: --cluster-dns="[]" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993257 2568 flags.go:64] FLAG: --cluster-domain="" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993260 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993262 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993265 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993268 2568 flags.go:64] FLAG: --container-log-max-files="5" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993272 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 01 00:33:40.997921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993275 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993278 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993281 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993284 2568 flags.go:64] FLAG: --contention-profiling="false" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993287 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993289 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993292 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993295 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993299 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993302 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993305 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993308 2568 flags.go:64] FLAG: --enable-load-reader="false" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993311 2568 flags.go:64] FLAG: --enable-server="true" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993313 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993321 2568 flags.go:64] FLAG: --event-burst="100" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993324 2568 flags.go:64] FLAG: --event-qps="50" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993327 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993330 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993333 2568 flags.go:64] FLAG: --eviction-hard="" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993337 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993339 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993343 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993345 2568 flags.go:64] FLAG: --eviction-soft="" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993348 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993351 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 01 00:33:40.998495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993355 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993358 2568 flags.go:64] FLAG: --experimental-mounter-path="" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993360 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993363 2568 flags.go:64] FLAG: --fail-swap-on="true" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993366 2568 flags.go:64] FLAG: --feature-gates="" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993370 2568 flags.go:64] FLAG: --file-check-frequency="20s" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993372 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993375 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993378 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993381 2568 flags.go:64] FLAG: --healthz-port="10248" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993386 2568 flags.go:64] FLAG: --help="false" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993389 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-137-42.ec2.internal" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993393 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993396 2568 flags.go:64] FLAG: --http-check-frequency="20s" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993399 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993402 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993405 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993409 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993412 2568 flags.go:64] FLAG: --image-service-endpoint="" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993414 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993418 2568 flags.go:64] FLAG: --kube-api-burst="100" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993421 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993424 2568 flags.go:64] FLAG: --kube-api-qps="50" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993427 2568 flags.go:64] FLAG: --kube-reserved="" Aug 01 00:33:40.999170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993429 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993432 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993435 2568 flags.go:64] FLAG: --kubelet-cgroups="" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993438 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993441 2568 flags.go:64] FLAG: --lock-file="" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993444 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993446 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993449 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993455 2568 flags.go:64] FLAG: --log-json-split-stream="false" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993461 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993464 2568 flags.go:64] FLAG: --log-text-split-stream="false" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993467 2568 flags.go:64] FLAG: --logging-format="text" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993469 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993473 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993476 2568 flags.go:64] FLAG: --manifest-url="" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993478 2568 flags.go:64] FLAG: --manifest-url-header="" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993483 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993486 2568 flags.go:64] FLAG: --max-open-files="1000000" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993491 2568 flags.go:64] FLAG: --max-pods="110" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993494 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993497 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993500 2568 flags.go:64] FLAG: --memory-manager-policy="None" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993502 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993505 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993508 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 01 00:33:40.999741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993511 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993520 2568 flags.go:64] FLAG: --node-status-max-images="50" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993523 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993526 2568 flags.go:64] FLAG: --oom-score-adj="-999" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993529 2568 flags.go:64] FLAG: --pod-cidr="" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993532 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993537 2568 flags.go:64] FLAG: --pod-manifest-path="" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993539 2568 flags.go:64] FLAG: --pod-max-pids="-1" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993542 2568 flags.go:64] FLAG: --pods-per-core="0" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993545 2568 flags.go:64] FLAG: --port="10250" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993548 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993551 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b2d8ac94674487a3" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993554 2568 flags.go:64] FLAG: --qos-reserved="" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993557 2568 flags.go:64] FLAG: --read-only-port="10255" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993560 2568 flags.go:64] FLAG: --register-node="true" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993563 2568 flags.go:64] FLAG: --register-with-taints="" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993566 2568 flags.go:64] FLAG: --registry-burst="10" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993574 2568 flags.go:64] FLAG: --registry-qps="5" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993577 2568 flags.go:64] FLAG: --reserved-cpus="" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993580 2568 flags.go:64] FLAG: --reserved-memory="" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993583 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993586 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993589 2568 flags.go:64] FLAG: --rotate-certificates="false" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993592 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993595 2568 flags.go:64] FLAG: --runonce="false" Aug 01 00:33:41.000388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993599 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993601 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993604 2568 flags.go:64] FLAG: --seccomp-default="false" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993607 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993610 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993613 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993616 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993619 2568 flags.go:64] FLAG: --storage-driver-password="root" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993622 2568 flags.go:64] FLAG: --storage-driver-secure="false" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993626 2568 flags.go:64] FLAG: --storage-driver-table="stats" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993629 2568 flags.go:64] FLAG: --storage-driver-user="root" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993631 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993634 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993637 2568 flags.go:64] FLAG: --system-cgroups="" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993640 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993645 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993648 2568 flags.go:64] FLAG: --tls-cert-file="" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993650 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993654 2568 flags.go:64] FLAG: --tls-min-version="" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993657 2568 flags.go:64] FLAG: --tls-private-key-file="" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993659 2568 flags.go:64] FLAG: --topology-manager-policy="none" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993662 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993665 2568 flags.go:64] FLAG: --topology-manager-scope="container" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993668 2568 flags.go:64] FLAG: --v="2" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993671 2568 flags.go:64] FLAG: --version="false" Aug 01 00:33:41.001098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993681 2568 flags.go:64] FLAG: --vmodule="" Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993685 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.993688 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993804 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993808 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993811 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993813 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993817 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993820 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993822 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993825 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993827 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993830 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993832 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993834 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993837 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993841 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993844 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993847 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993849 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:33:41.001684 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993852 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993854 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993857 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993860 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993862 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993865 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993868 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993870 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993873 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993875 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993878 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993880 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993883 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993893 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993897 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993900 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993903 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993920 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993923 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:33:41.002178 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993927 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993931 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993934 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993936 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993939 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993941 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993944 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993946 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993949 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993953 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993956 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993959 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993961 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993964 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993966 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993969 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993972 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993974 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993977 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993979 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:33:41.002662 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993982 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993984 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993987 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993989 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993991 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993994 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.993996 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994000 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994003 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994005 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994008 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994010 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994016 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994018 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994021 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994024 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994026 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994028 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994031 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:33:41.003150 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994033 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994037 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994042 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994044 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994047 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994049 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994052 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994055 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994057 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994060 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994062 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994065 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994067 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994070 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994072 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994075 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994078 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994080 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994084 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:33:41.003615 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994086 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994089 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994092 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994094 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:40.994097 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:40.994863 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.002282 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.002303 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.002325 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.002331 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002387 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002392 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002395 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002398 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002401 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:33:41.004106 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002404 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002407 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002410 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002413 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002415 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002418 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002420 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002423 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002426 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002428 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002431 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002434 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002436 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002438 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002441 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002443 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002446 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002448 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002451 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002453 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:33:41.004483 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002456 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002460 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002464 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002467 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002471 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002473 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002476 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002478 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002482 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002485 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002489 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002492 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002495 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002498 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002501 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002504 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002506 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002509 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:33:41.004991 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002512 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002515 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002517 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002520 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002522 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002525 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002527 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002530 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002532 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002535 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002537 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002539 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002542 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002545 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002547 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002550 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002553 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002556 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002559 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002562 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:33:41.005422 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002564 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002567 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002569 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002572 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002575 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002577 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002580 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002582 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002585 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002587 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002589 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002592 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002594 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002597 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002600 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002602 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002604 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002607 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002609 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002611 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:33:41.005954 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002614 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002616 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002618 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002621 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002623 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002626 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002628 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002630 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002633 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002636 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002639 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002641 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002644 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002647 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002649 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:33:41.006434 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002652 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.002657 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002771 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002775 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002779 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002781 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002784 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002787 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002790 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002792 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002795 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002797 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002800 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002802 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:33:41.006790 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002805 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002807 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002810 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002812 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002815 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002818 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002820 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002823 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002825 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002827 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002830 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002832 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002836 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002840 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002843 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002845 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002848 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002851 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002853 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002856 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:33:41.007201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002858 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002861 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002864 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002867 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002869 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002872 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002874 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002877 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002879 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002882 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002884 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002886 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002889 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002891 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002893 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002896 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002898 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002901 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002903 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:33:41.007677 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002922 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002925 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002928 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002931 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002933 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002936 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002938 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002941 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002944 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002946 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002950 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002954 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002957 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002960 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002963 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002966 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002969 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002971 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002974 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002976 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:33:41.008161 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002979 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002981 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002984 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002986 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002988 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002991 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002993 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002995 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.002998 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003000 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003003 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003005 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003007 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003010 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003012 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003015 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003017 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003020 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003022 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003025 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:33:41.008634 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003027 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003030 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003032 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003035 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003037 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003040 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003043 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:41.003045 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.003050 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.003177 2568 server.go:962] "Client rotation is on, will bootstrap in background" Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.005379 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.006821 2568 server.go:1019] "Starting client certificate rotation" Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.006924 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 01 00:33:41.009200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.006961 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 01 00:33:41.009517 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.009225 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 01 00:33:41.013294 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.013268 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 01 00:33:41.023176 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.023150 2568 log.go:25] "Validated CRI v1 runtime API" Aug 01 00:33:41.025525 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.025505 2568 log.go:25] "Validated CRI v1 image API" Aug 01 00:33:41.026399 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.026382 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 01 00:33:41.031496 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.031476 2568 fs.go:135] Filesystem UUIDs: map[4aefa8dc-778d-415b-b111-e2556a26cd9a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b4f22153-22b4-4b01-af95-63ecec966fc6:/dev/nvme0n1p4] Aug 01 00:33:41.031553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.031497 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 01 00:33:41.036927 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.036790 2568 manager.go:216] Machine: {Timestamp:2026-08-01 00:33:41.035489527 +0000 UTC m=+0.299421220 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106772 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eb5db061726bab1f3eb46d31b5902 SystemUUID:ec2eb5db-0617-26ba-b1f3-eb46d31b5902 BootID:8c167663-8c3a-4e41-a834-713027899ca4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9f:8c:45:a7:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9f:8c:45:a7:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:d2:d4:59:98:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 01 00:33:41.036927 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.036895 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 01 00:33:41.037076 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.036998 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 01 00:33:41.037173 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.037157 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 01 00:33:41.039784 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.039755 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 01 00:33:41.039946 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.039788 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-42.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 01 00:33:41.040571 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.040560 2568 topology_manager.go:138] "Creating topology manager with none policy" Aug 01 00:33:41.040606 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.040573 2568 container_manager_linux.go:309] "Creating device plugin manager" Aug 01 00:33:41.040606 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.040585 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 01 00:33:41.041247 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.041237 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 01 00:33:41.042253 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.042243 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 01 00:33:41.047014 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047001 2568 state_mem.go:36] "Initialized new in-memory state store" Aug 01 00:33:41.047138 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047128 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 01 00:33:41.047167 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047161 2568 kubelet.go:486] "Attempting to sync node with API server" Aug 01 00:33:41.047200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047175 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 01 00:33:41.047200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047190 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 01 00:33:41.047200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047199 2568 kubelet.go:398] "Adding apiserver pod source" Aug 01 00:33:41.047282 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047209 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 01 00:33:41.047571 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047560 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 01 00:33:41.047610 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.047578 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 01 00:33:41.050194 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.050177 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 01 00:33:41.052890 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.052863 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 01 00:33:41.052992 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.052937 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 01 00:33:41.054982 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.054963 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 01 00:33:41.054982 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.054983 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.054991 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055001 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055013 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055019 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055027 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055032 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055039 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055044 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055053 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055058 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055084 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 01 00:33:41.055120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.055092 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 01 00:33:41.058454 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.058439 2568 server.go:1268] "Started kubelet" Aug 01 00:33:41.058578 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.058527 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 01 00:33:41.058769 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.058525 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 01 00:33:41.058870 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.058853 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 01 00:33:41.059392 ip-10-0-137-42 systemd[1]: Started Kubernetes Kubelet. Aug 01 00:33:41.060288 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.060194 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 01 00:33:41.060626 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.060612 2568 server.go:310] "Adding debug handlers to kubelet server" Aug 01 00:33:41.062401 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.062382 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-42.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 01 00:33:41.062480 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.062402 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-42.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 01 00:33:41.062543 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.062520 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 01 00:33:41.063361 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.062544 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-42.ec2.internal.18c78730d3dde590 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-42.ec2.internal,UID:ip-10-0-137-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-42.ec2.internal,},FirstTimestamp:2026-08-01 00:33:41.058405776 +0000 UTC m=+0.322337470,LastTimestamp:2026-08-01 00:33:41.058405776 +0000 UTC m=+0.322337470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-42.ec2.internal,}" Aug 01 00:33:41.064541 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.064506 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 01 00:33:41.065744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.065731 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 01 00:33:41.065803 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.065745 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 01 00:33:41.065853 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.065824 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.067275 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.067254 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 01 00:33:41.067361 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.067278 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 01 00:33:41.067361 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.067254 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 01 00:33:41.067499 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.067400 2568 reconstruct.go:97] "Volume reconstruction finished" Aug 01 00:33:41.067499 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.067407 2568 reconciler.go:29] "Reconciler: start to sync state" Aug 01 00:33:41.067642 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.067603 2568 factory.go:55] Registering systemd factory Aug 01 00:33:41.067642 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.067634 2568 factory.go:223] Registration of the systemd container factory successfully Aug 01 00:33:41.067846 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.067802 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 01 00:33:41.067989 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.067968 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-42.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 01 00:33:41.068235 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.068221 2568 factory.go:153] Registering CRI-O factory Aug 01 00:33:41.068294 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.068239 2568 factory.go:223] Registration of the crio container factory successfully Aug 01 00:33:41.068340 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.068293 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 01 00:33:41.068340 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.068322 2568 factory.go:103] Registering Raw factory Aug 01 00:33:41.068340 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.068339 2568 manager.go:1196] Started watching for new ooms in manager Aug 01 00:33:41.069173 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.069160 2568 manager.go:349] Starting recovery of all containers Aug 01 00:33:41.080803 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.080640 2568 manager.go:354] Recovery completed Aug 01 00:33:41.082577 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.082556 2568 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Aug 01 00:33:41.083194 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.083176 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tc8jk" Aug 01 00:33:41.085631 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.085618 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:33:41.087646 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.087632 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:33:41.087713 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.087660 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:33:41.087713 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.087673 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:33:41.088288 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.088275 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:33:41.088288 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.088277 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 01 00:33:41.088398 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.088293 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 01 00:33:41.088398 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.088309 2568 state_mem.go:36] "Initialized new in-memory state store" Aug 01 00:33:41.088964 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.088948 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:33:41.089069 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.088978 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:33:41.089069 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.088992 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:33:41.090407 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.090391 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tc8jk" Aug 01 00:33:41.091520 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.091508 2568 policy_none.go:49] "None policy: Start" Aug 01 00:33:41.091575 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.091525 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 01 00:33:41.091575 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.091536 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 01 00:33:41.093622 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.093609 2568 policy_none.go:47] "Start" Aug 01 00:33:41.096612 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.096542 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-42.ec2.internal.18c78730d59c176b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-42.ec2.internal,UID:ip-10-0-137-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-42.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-42.ec2.internal,},FirstTimestamp:2026-08-01 00:33:41.087647595 +0000 UTC m=+0.351579289,LastTimestamp:2026-08-01 00:33:41.087647595 +0000 UTC m=+0.351579289,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-42.ec2.internal,}" Aug 01 00:33:41.131633 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.131611 2568 manager.go:337] "Starting Device Plugin manager" Aug 01 00:33:41.131728 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.131648 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 01 00:33:41.131728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.131658 2568 server.go:87] "Starting device plugin registration server" Aug 01 00:33:41.132059 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.132037 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 01 00:33:41.132158 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.132055 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 01 00:33:41.132213 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.132177 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 01 00:33:41.132298 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.132280 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 01 00:33:41.132298 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.132297 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 01 00:33:41.132891 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.132869 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 01 00:33:41.132985 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.132949 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.168297 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.168262 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 01 00:33:41.173099 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.169594 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 01 00:33:41.173099 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.169660 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 01 00:33:41.173099 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.169687 2568 kubelet.go:2448] "Starting kubelet main sync loop" Aug 01 00:33:41.173099 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.169727 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 01 00:33:41.176080 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.176055 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:33:41.232393 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.232357 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:33:41.233334 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.233271 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:33:41.233334 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.233310 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:33:41.233334 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.233328 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:33:41.233525 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.233357 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.243472 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.243450 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.243561 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.243475 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-42.ec2.internal\": node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.270710 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.270663 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal"] Aug 01 00:33:41.270859 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.270740 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:33:41.272224 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.272206 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:33:41.272329 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.272241 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:33:41.272329 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.272257 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:33:41.274609 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.274593 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:33:41.274729 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.274713 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.274773 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.274750 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:33:41.274967 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.274953 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.275352 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.275335 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:33:41.275398 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.275335 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:33:41.275398 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.275396 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:33:41.275462 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.275408 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:33:41.275462 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.275371 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:33:41.275462 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.275447 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:33:41.277688 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.277672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.277745 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.277706 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:33:41.278482 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.278464 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:33:41.278549 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.278512 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:33:41.278549 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.278527 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:33:41.306758 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.306735 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-42.ec2.internal\" not found" node="ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.311342 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.311326 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-42.ec2.internal\" not found" node="ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.368969 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.368937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7ccfef67ed5bb286014d58f5d8b58af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal\" (UID: \"e7ccfef67ed5bb286014d58f5d8b58af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.368969 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.368969 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7ccfef67ed5bb286014d58f5d8b58af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal\" (UID: \"e7ccfef67ed5bb286014d58f5d8b58af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.369125 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.368985 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d78001921e659c8098fde0e8ca179c5e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-42.ec2.internal\" (UID: \"d78001921e659c8098fde0e8ca179c5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.375526 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.375503 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.470028 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.469986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d78001921e659c8098fde0e8ca179c5e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-42.ec2.internal\" (UID: \"d78001921e659c8098fde0e8ca179c5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.470028 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.470036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7ccfef67ed5bb286014d58f5d8b58af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal\" (UID: \"e7ccfef67ed5bb286014d58f5d8b58af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.470150 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.470053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7ccfef67ed5bb286014d58f5d8b58af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal\" (UID: \"e7ccfef67ed5bb286014d58f5d8b58af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.470150 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.470052 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d78001921e659c8098fde0e8ca179c5e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-42.ec2.internal\" (UID: \"d78001921e659c8098fde0e8ca179c5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.470150 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.470073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7ccfef67ed5bb286014d58f5d8b58af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal\" (UID: \"e7ccfef67ed5bb286014d58f5d8b58af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.470150 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.470082 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7ccfef67ed5bb286014d58f5d8b58af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal\" (UID: \"e7ccfef67ed5bb286014d58f5d8b58af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.476109 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.476086 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.577002 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.576936 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.610148 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.610113 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.614283 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.614247 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" Aug 01 00:33:41.677767 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.677735 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.778331 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.778293 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.878987 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.878899 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.979593 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:41.979566 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-42.ec2.internal\" not found" Aug 01 00:33:41.987616 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:41.987600 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:33:42.006468 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:42.006444 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 01 00:33:42.006599 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:42.006570 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 01 00:33:42.006645 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:42.006585 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 01 00:33:43.102059 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.102035 2568 apiserver.go:52] "Watching apiserver" Aug 01 00:33:43.118694 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.118664 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 01 00:33:43.118999 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.118977 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-249m9","openshift-network-operator/iptables-alerter-j4kdm","openshift-ovn-kubernetes/ovnkube-node-ptg24","openshift-cluster-node-tuning-operator/tuned-npmz6","openshift-dns/node-resolver-x4mkf","openshift-multus/multus-6sqf4","openshift-network-diagnostics/network-check-target-fgkhq","kube-system/global-pull-secret-syncer-d85hp","kube-system/konnectivity-agent-75wgw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh","openshift-image-registry/node-ca-vm2k4","openshift-multus/multus-additional-cni-plugins-4khxh"] Aug 01 00:33:43.120416 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.120397 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 01 00:33:43.120416 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.120410 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" Aug 01 00:33:43.122093 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.122077 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:43.122191 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.122136 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:43.124645 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.124626 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.126871 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.126853 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 01 00:33:43.126871 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.126866 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xjlc6\"" Aug 01 00:33:43.127067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.126871 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:33:43.127067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.126852 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 01 00:33:43.127366 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.127348 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.127825 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.127797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00392c01-c744-4f1f-be3a-6eafbc64af50-host-slash\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.127873 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.127840 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94tr\" (UniqueName: \"kubernetes.io/projected/00392c01-c744-4f1f-be3a-6eafbc64af50-kube-api-access-r94tr\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.127873 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.127862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:43.127936 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.127899 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdfs\" (UniqueName: \"kubernetes.io/projected/dcf38574-98e0-464c-8f21-bdb577e4b0ec-kube-api-access-mkdfs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:43.127987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.127945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00392c01-c744-4f1f-be3a-6eafbc64af50-iptables-alerter-script\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.128136 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.128123 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:33:43.129427 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.129409 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fw626\"" Aug 01 00:33:43.129537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.129417 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 01 00:33:43.129537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.129474 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 01 00:33:43.129640 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.129576 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 01 00:33:43.129640 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.129585 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 01 00:33:43.129640 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.129593 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 01 00:33:43.129840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.129825 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 01 00:33:43.129875 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.129855 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 01 00:33:43.130501 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.130487 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.132395 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.132377 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:33:43.132395 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.132394 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 01 00:33:43.132520 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.132493 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6lfmv\"" Aug 01 00:33:43.132680 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.132664 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.134598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.134584 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 01 00:33:43.134688 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.134620 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 01 00:33:43.134688 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.134656 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 01 00:33:43.134787 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.134714 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bn95k\"" Aug 01 00:33:43.134840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.134815 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" Aug 01 00:33:43.134914 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.134898 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.136579 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.136561 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 01 00:33:43.136698 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.136678 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 01 00:33:43.137011 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.136995 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 01 00:33:43.137092 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.137075 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9dtsv\"" Aug 01 00:33:43.137170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.137157 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:43.138052 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.137963 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 01 00:33:43.138359 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.137234 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:33:43.141870 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.141851 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.141946 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.141930 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:43.144149 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.144131 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:33:43.145070 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.145049 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 01 00:33:43.146598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.146469 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 01 00:33:43.146598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.146534 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.146598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.146557 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kvdvw\"" Aug 01 00:33:43.146598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.146577 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 01 00:33:43.148843 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.148671 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 01 00:33:43.149463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.148913 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2x8zp\"" Aug 01 00:33:43.149463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.148936 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 01 00:33:43.149463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.149047 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 01 00:33:43.149463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.149261 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.149616 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.149512 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-31 00:28:42 +0000 UTC" deadline="2028-05-08 03:17:36.709750502 +0000 UTC" Aug 01 00:33:43.149616 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.149559 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15506h43m53.560193402s" Aug 01 00:33:43.151173 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.151155 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 01 00:33:43.151173 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.151168 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 01 00:33:43.151345 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.151272 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal"] Aug 01 00:33:43.151345 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.151292 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-42vns\"" Aug 01 00:33:43.151345 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.151307 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal"] Aug 01 00:33:43.151517 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.151390 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.151578 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.151565 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 01 00:33:43.153112 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.153098 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7gp5x\"" Aug 01 00:33:43.153200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.153102 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 01 00:33:43.153405 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.153390 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 01 00:33:43.154798 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.154780 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gc8dl" Aug 01 00:33:43.163523 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.163504 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gc8dl" Aug 01 00:33:43.222492 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.222466 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 01 00:33:43.228110 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228090 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-daemon-config\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228254 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228117 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-log-socket\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.228254 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228133 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysconfig\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.228254 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228150 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-cni-bin\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228254 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-cni-multus\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228254 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228193 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.228254 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228234 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkts\" (UniqueName: \"kubernetes.io/projected/10cbd086-8ace-48e5-bda2-915b5ba02111-kube-api-access-9nkts\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.228254 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fb3d89c-6d48-445c-9726-4b33c252e881-cni-binary-copy\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovnkube-config\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.228463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-tuned\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.228463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228324 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-cni-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228342 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-cnibin\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228355 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-os-release\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228368 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.228463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdfs\" (UniqueName: \"kubernetes.io/projected/dcf38574-98e0-464c-8f21-bdb577e4b0ec-kube-api-access-mkdfs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:43.228667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228461 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00392c01-c744-4f1f-be3a-6eafbc64af50-iptables-alerter-script\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.228667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.228667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-systemd\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.228667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228604 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89569cc7-3743-47cf-b288-2dac9c3e5f01-tmp\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.228667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.228809 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdf2\" (UniqueName: \"kubernetes.io/projected/78d38db1-2bcf-4778-8227-2956eec1f6f0-kube-api-access-4xdf2\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.228809 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228691 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cbd086-8ace-48e5-bda2-915b5ba02111-tmp-dir\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.228809 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-kubernetes\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.228809 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228744 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.228809 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228760 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-registration-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.228809 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228785 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-dbus\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-cni-bin\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-modprobe-d\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228860 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228885 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-host\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-conf-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228923 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-device-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228943 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-k8s-cni-cncf-io\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.228987 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228973 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.228991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-kubelet\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229017 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00392c01-c744-4f1f-be3a-6eafbc64af50-iptables-alerter-script\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229043 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-slash\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229068 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-etc-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-env-overrides\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229099 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovnkube-script-lib\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysctl-conf\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229140 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-multus-certs\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmx2\" (UniqueName: \"kubernetes.io/projected/8fb3d89c-6d48-445c-9726-4b33c252e881-kube-api-access-hpmx2\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229172 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-hostroot\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229187 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229227 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ml7\" (UniqueName: \"kubernetes.io/projected/89569cc7-3743-47cf-b288-2dac9c3e5f01-kube-api-access-v4ml7\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229249 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnxw\" (UniqueName: \"kubernetes.io/projected/8afb46d3-d50e-4f0c-b804-1b33b989534e-kube-api-access-6vnxw\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.229258 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229263 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7zz\" (UniqueName: \"kubernetes.io/projected/06e31806-756d-4782-9193-ac8ce656d17b-kube-api-access-5l7zz\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-system-cni-dir\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229305 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a-agent-certs\") pod \"konnectivity-agent-75wgw\" (UID: \"fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a\") " pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-socket-dir-parent\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-netns\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229375 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-cni-netd\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229398 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rmf\" (UniqueName: \"kubernetes.io/projected/0007ab3a-bc9a-4a12-9a80-baee7909713c-kube-api-access-55rmf\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-var-lib-kubelet\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229444 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94tr\" (UniqueName: \"kubernetes.io/projected/00392c01-c744-4f1f-be3a-6eafbc64af50-kube-api-access-r94tr\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229472 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-run-netns\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229514 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06e31806-756d-4782-9193-ac8ce656d17b-host\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-cnibin\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229544 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-systemd-units\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-var-lib-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229608 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-node-log\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229624 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovn-node-metrics-cert\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-sys\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.229671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-etc-kubernetes\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-socket-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229691 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-sys-fs\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229705 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-ovn\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229719 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-os-release\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a-konnectivity-ca\") pod \"konnectivity-agent-75wgw\" (UID: \"fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a\") " pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229794 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229819 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysctl-d\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229853 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-run\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.229854 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.229931 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:33:43.729915007 +0000 UTC m=+1.939543441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-lib-modules\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229960 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10cbd086-8ace-48e5-bda2-915b5ba02111-hosts-file\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-kubelet\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.229996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00392c01-c744-4f1f-be3a-6eafbc64af50-host-slash\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.230157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.230010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-systemd\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.230633 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.230030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-system-cni-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.230633 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.230045 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06e31806-756d-4782-9193-ac8ce656d17b-serviceca\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.230633 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.230055 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00392c01-c744-4f1f-be3a-6eafbc64af50-host-slash\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.230633 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.230070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-kubelet-config\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.234341 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.234314 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 01 00:33:43.237766 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.237744 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94tr\" (UniqueName: \"kubernetes.io/projected/00392c01-c744-4f1f-be3a-6eafbc64af50-kube-api-access-r94tr\") pod \"iptables-alerter-j4kdm\" (UID: \"00392c01-c744-4f1f-be3a-6eafbc64af50\") " pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.237766 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.237760 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdfs\" (UniqueName: \"kubernetes.io/projected/dcf38574-98e0-464c-8f21-bdb577e4b0ec-kube-api-access-mkdfs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:43.273147 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.273130 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:33:43.331050 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331013 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysconfig\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.331050 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-cni-bin\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-cni-multus\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331140 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-cni-bin\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkts\" (UniqueName: \"kubernetes.io/projected/10cbd086-8ace-48e5-bda2-915b5ba02111-kube-api-access-9nkts\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fb3d89c-6d48-445c-9726-4b33c252e881-cni-binary-copy\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovnkube-config\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331143 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-cni-multus\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331143 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysconfig\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-tuned\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-cni-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331342 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-cni-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331388 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-cnibin\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331463 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-os-release\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331482 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-cnibin\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331506 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331547 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-os-release\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-systemd\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331619 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-systemd\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331642 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89569cc7-3743-47cf-b288-2dac9c3e5f01-tmp\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdf2\" (UniqueName: \"kubernetes.io/projected/78d38db1-2bcf-4778-8227-2956eec1f6f0-kube-api-access-4xdf2\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cbd086-8ace-48e5-bda2-915b5ba02111-tmp-dir\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331793 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-kubernetes\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.331840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331867 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331888 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fb3d89c-6d48-445c-9726-4b33c252e881-cni-binary-copy\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-registration-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331921 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovnkube-config\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331934 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-dbus\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331969 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-cni-bin\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331988 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-dbus\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.331996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-modprobe-d\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332018 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332037 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-cni-bin\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-kubernetes\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-host\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-conf-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-device-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332121 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cbd086-8ace-48e5-bda2-915b5ba02111-tmp-dir\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.332553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332156 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-conf-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-k8s-cni-cncf-io\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332201 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-host\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332233 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332291 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-k8s-cni-cncf-io\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332308 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-modprobe-d\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332329 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-kubelet\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332346 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-device-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-slash\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332372 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-kubelet\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332408 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-slash\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.332419 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332434 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332441 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-etc-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.333339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332435 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-registration-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332456 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78d38db1-2bcf-4778-8227-2956eec1f6f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.332478 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret podName:d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f nodeName:}" failed. No retries permitted until 2026-08-01 00:33:43.832460938 +0000 UTC m=+2.042089380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret") pod "global-pull-secret-syncer-d85hp" (UID: "d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332531 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-etc-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-env-overrides\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovnkube-script-lib\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysctl-conf\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-multus-certs\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmx2\" (UniqueName: \"kubernetes.io/projected/8fb3d89c-6d48-445c-9726-4b33c252e881-kube-api-access-hpmx2\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332682 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-multus-certs\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysctl-conf\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-hostroot\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332836 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-hostroot\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332943 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ml7\" (UniqueName: \"kubernetes.io/projected/89569cc7-3743-47cf-b288-2dac9c3e5f01-kube-api-access-v4ml7\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnxw\" (UniqueName: \"kubernetes.io/projected/8afb46d3-d50e-4f0c-b804-1b33b989534e-kube-api-access-6vnxw\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.334156 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.332997 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7zz\" (UniqueName: \"kubernetes.io/projected/06e31806-756d-4782-9193-ac8ce656d17b-kube-api-access-5l7zz\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-system-cni-dir\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333039 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a-agent-certs\") pod \"konnectivity-agent-75wgw\" (UID: \"fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a\") " pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-socket-dir-parent\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333091 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovnkube-script-lib\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-netns\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-cni-netd\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333234 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-system-cni-dir\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333250 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0007ab3a-bc9a-4a12-9a80-baee7909713c-env-overrides\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rmf\" (UniqueName: \"kubernetes.io/projected/0007ab3a-bc9a-4a12-9a80-baee7909713c-kube-api-access-55rmf\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333314 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-run-netns\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333375 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-cni-netd\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333380 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-var-lib-kubelet\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333418 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-run-netns\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06e31806-756d-4782-9193-ac8ce656d17b-host\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333449 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-var-lib-kubelet\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.334796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-cnibin\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-host-run-netns\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333484 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-socket-dir-parent\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333492 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-systemd-units\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333534 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-cnibin\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-systemd-units\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-var-lib-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333584 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-var-lib-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-node-log\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333640 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06e31806-756d-4782-9193-ac8ce656d17b-host\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovn-node-metrics-cert\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333684 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-node-log\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-sys\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-etc-kubernetes\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333723 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-sys\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333739 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-socket-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333764 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-etc-kubernetes\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333766 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-sys-fs\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.335323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333794 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-ovn\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333812 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-sys-fs\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333843 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-ovn\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-os-release\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333870 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78d38db1-2bcf-4778-8227-2956eec1f6f0-os-release\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a-konnectivity-ca\") pod \"konnectivity-agent-75wgw\" (UID: \"fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a\") " pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8afb46d3-d50e-4f0c-b804-1b33b989534e-socket-dir\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysctl-d\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-openvswitch\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.333982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-run\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334072 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-tuned\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334103 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-etc-sysctl-d\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334129 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-lib-modules\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334152 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-run\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10cbd086-8ace-48e5-bda2-915b5ba02111-hosts-file\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-kubelet\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10cbd086-8ace-48e5-bda2-915b5ba02111-hosts-file\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.336065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334290 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-systemd\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334316 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89569cc7-3743-47cf-b288-2dac9c3e5f01-lib-modules\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334308 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-host-var-lib-kubelet\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334360 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-system-cni-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06e31806-756d-4782-9193-ac8ce656d17b-serviceca\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-run-systemd\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334409 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-kubelet-config\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334417 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a-konnectivity-ca\") pod \"konnectivity-agent-75wgw\" (UID: \"fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a\") " pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-daemon-config\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334447 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fb3d89c-6d48-445c-9726-4b33c252e881-system-cni-dir\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334463 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-log-socket\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334492 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-kubelet-config\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334527 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0007ab3a-bc9a-4a12-9a80-baee7909713c-log-socket\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334660 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89569cc7-3743-47cf-b288-2dac9c3e5f01-tmp\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334846 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fb3d89c-6d48-445c-9726-4b33c252e881-multus-daemon-config\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.334999 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06e31806-756d-4782-9193-ac8ce656d17b-serviceca\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.336114 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0007ab3a-bc9a-4a12-9a80-baee7909713c-ovn-node-metrics-cert\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.336833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.336347 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a-agent-certs\") pod \"konnectivity-agent-75wgw\" (UID: \"fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a\") " pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:33:43.339383 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.339364 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:33:43.339474 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.339387 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:33:43.339474 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.339400 2568 projected.go:196] Error preparing data for projected volume kube-api-access-tbtth for pod openshift-network-diagnostics/network-check-target-fgkhq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:43.339474 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.339465 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth podName:ef0430f8-146f-411e-9581-b62cf7e57a76 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:43.839446655 +0000 UTC m=+2.049075104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tbtth" (UniqueName: "kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth") pod "network-check-target-fgkhq" (UID: "ef0430f8-146f-411e-9581-b62cf7e57a76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:43.342242 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.342163 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ml7\" (UniqueName: \"kubernetes.io/projected/89569cc7-3743-47cf-b288-2dac9c3e5f01-kube-api-access-v4ml7\") pod \"tuned-npmz6\" (UID: \"89569cc7-3743-47cf-b288-2dac9c3e5f01\") " pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.344924 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.342381 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkts\" (UniqueName: \"kubernetes.io/projected/10cbd086-8ace-48e5-bda2-915b5ba02111-kube-api-access-9nkts\") pod \"node-resolver-x4mkf\" (UID: \"10cbd086-8ace-48e5-bda2-915b5ba02111\") " pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.344924 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.342548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdf2\" (UniqueName: \"kubernetes.io/projected/78d38db1-2bcf-4778-8227-2956eec1f6f0-kube-api-access-4xdf2\") pod \"multus-additional-cni-plugins-4khxh\" (UID: \"78d38db1-2bcf-4778-8227-2956eec1f6f0\") " pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.344924 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.342621 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmx2\" (UniqueName: \"kubernetes.io/projected/8fb3d89c-6d48-445c-9726-4b33c252e881-kube-api-access-hpmx2\") pod \"multus-6sqf4\" (UID: \"8fb3d89c-6d48-445c-9726-4b33c252e881\") " pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.344924 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.342896 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnxw\" (UniqueName: \"kubernetes.io/projected/8afb46d3-d50e-4f0c-b804-1b33b989534e-kube-api-access-6vnxw\") pod \"aws-ebs-csi-driver-node-mf7nh\" (UID: \"8afb46d3-d50e-4f0c-b804-1b33b989534e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.344924 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.343603 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7zz\" (UniqueName: \"kubernetes.io/projected/06e31806-756d-4782-9193-ac8ce656d17b-kube-api-access-5l7zz\") pod \"node-ca-vm2k4\" (UID: \"06e31806-756d-4782-9193-ac8ce656d17b\") " pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.345588 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.345570 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rmf\" (UniqueName: \"kubernetes.io/projected/0007ab3a-bc9a-4a12-9a80-baee7909713c-kube-api-access-55rmf\") pod \"ovnkube-node-ptg24\" (UID: \"0007ab3a-bc9a-4a12-9a80-baee7909713c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.458194 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.458107 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j4kdm" Aug 01 00:33:43.465794 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.465772 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:33:43.468598 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:43.468552 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00392c01_c744_4f1f_be3a_6eafbc64af50.slice/crio-0ad9611c7a7f0a0f2ecc78951adfaf6af0662735fe1f54696685b5bdb2775bdb WatchSource:0}: Error finding container 0ad9611c7a7f0a0f2ecc78951adfaf6af0662735fe1f54696685b5bdb2775bdb: Status 404 returned error can't find the container with id 0ad9611c7a7f0a0f2ecc78951adfaf6af0662735fe1f54696685b5bdb2775bdb Aug 01 00:33:43.475084 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:43.475060 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0007ab3a_bc9a_4a12_9a80_baee7909713c.slice/crio-b554c01b9aecad6d72258b6928004abcbff627ccc7cf1c9a70417ff9b3c22acf WatchSource:0}: Error finding container b554c01b9aecad6d72258b6928004abcbff627ccc7cf1c9a70417ff9b3c22acf: Status 404 returned error can't find the container with id b554c01b9aecad6d72258b6928004abcbff627ccc7cf1c9a70417ff9b3c22acf Aug 01 00:33:43.481802 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.481782 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-npmz6" Aug 01 00:33:43.485164 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.485144 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x4mkf" Aug 01 00:33:43.494545 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:43.494520 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cbd086_8ace_48e5_bda2_915b5ba02111.slice/crio-eab03267b578581332b1f6476da0c009b1c80ae188fa350ff889703dba97965a WatchSource:0}: Error finding container eab03267b578581332b1f6476da0c009b1c80ae188fa350ff889703dba97965a: Status 404 returned error can't find the container with id eab03267b578581332b1f6476da0c009b1c80ae188fa350ff889703dba97965a Aug 01 00:33:43.514415 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.514394 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6sqf4" Aug 01 00:33:43.521337 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.521317 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:33:43.523357 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:43.523328 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb3d89c_6d48_445c_9726_4b33c252e881.slice/crio-0262b7e2e0dc077eaff5d0b939839a5de694a2af734dd2b25e9e202f4d5506b3 WatchSource:0}: Error finding container 0262b7e2e0dc077eaff5d0b939839a5de694a2af734dd2b25e9e202f4d5506b3: Status 404 returned error can't find the container with id 0262b7e2e0dc077eaff5d0b939839a5de694a2af734dd2b25e9e202f4d5506b3 Aug 01 00:33:43.530283 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:43.530259 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6a6f9e_b36a_45e7_9f1a_b8bf821dc03a.slice/crio-1abe03ead35b58a37577a85e7ad55512d738c4bcf53596a92a5ead163a51185a WatchSource:0}: Error finding container 1abe03ead35b58a37577a85e7ad55512d738c4bcf53596a92a5ead163a51185a: Status 404 returned error can't find the container with id 1abe03ead35b58a37577a85e7ad55512d738c4bcf53596a92a5ead163a51185a Aug 01 00:33:43.536601 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.536583 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" Aug 01 00:33:43.544897 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:43.544877 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afb46d3_d50e_4f0c_b804_1b33b989534e.slice/crio-4894644ff5f102f1503859ab35fd46d6d53507f72a21ca017d391b70732b3f7c WatchSource:0}: Error finding container 4894644ff5f102f1503859ab35fd46d6d53507f72a21ca017d391b70732b3f7c: Status 404 returned error can't find the container with id 4894644ff5f102f1503859ab35fd46d6d53507f72a21ca017d391b70732b3f7c Aug 01 00:33:43.561222 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.561189 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vm2k4" Aug 01 00:33:43.566030 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.566012 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4khxh" Aug 01 00:33:43.569062 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:43.569038 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e31806_756d_4782_9193_ac8ce656d17b.slice/crio-85e461d00a93bebefb101c3c756f86b9f0a03b1d6ca004775b30e3650f0da4f7 WatchSource:0}: Error finding container 85e461d00a93bebefb101c3c756f86b9f0a03b1d6ca004775b30e3650f0da4f7: Status 404 returned error can't find the container with id 85e461d00a93bebefb101c3c756f86b9f0a03b1d6ca004775b30e3650f0da4f7 Aug 01 00:33:43.574378 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:33:43.574359 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d38db1_2bcf_4778_8227_2956eec1f6f0.slice/crio-ddd7fecff8dfdbfdc1c2428d509d7e04adafd78088ff8734408b3f81132f6f87 WatchSource:0}: Error finding container ddd7fecff8dfdbfdc1c2428d509d7e04adafd78088ff8734408b3f81132f6f87: Status 404 returned error can't find the container with id ddd7fecff8dfdbfdc1c2428d509d7e04adafd78088ff8734408b3f81132f6f87 Aug 01 00:33:43.656289 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.656252 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:33:43.738053 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.737460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:43.738053 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.737664 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:43.738053 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.737732 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:33:44.737711682 +0000 UTC m=+2.947340144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:43.837912 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.837879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:43.838153 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.838043 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:43.838153 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.838123 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret podName:d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f nodeName:}" failed. No retries permitted until 2026-08-01 00:33:44.838102478 +0000 UTC m=+3.047730931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret") pod "global-pull-secret-syncer-d85hp" (UID: "d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:43.938918 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.938880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:43.939228 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.939190 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:33:43.939277 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.939237 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:33:43.939277 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.939251 2568 projected.go:196] Error preparing data for projected volume kube-api-access-tbtth for pod openshift-network-diagnostics/network-check-target-fgkhq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:43.939356 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:43.939312 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth podName:ef0430f8-146f-411e-9581-b62cf7e57a76 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:44.939294521 +0000 UTC m=+3.148922970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbtth" (UniqueName: "kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth") pod "network-check-target-fgkhq" (UID: "ef0430f8-146f-411e-9581-b62cf7e57a76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:43.959905 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:43.959880 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:33:44.166823 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.166702 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-31 00:28:43 +0000 UTC" deadline="2028-03-14 08:23:25.421804469 +0000 UTC" Aug 01 00:33:44.166823 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.166756 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14191h49m41.255052418s" Aug 01 00:33:44.229623 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.229061 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:44.229623 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.229180 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:44.232199 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.232104 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerStarted","Data":"ddd7fecff8dfdbfdc1c2428d509d7e04adafd78088ff8734408b3f81132f6f87"} Aug 01 00:33:44.235147 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.235021 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" event={"ID":"8afb46d3-d50e-4f0c-b804-1b33b989534e","Type":"ContainerStarted","Data":"4894644ff5f102f1503859ab35fd46d6d53507f72a21ca017d391b70732b3f7c"} Aug 01 00:33:44.250800 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.249913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-75wgw" event={"ID":"fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a","Type":"ContainerStarted","Data":"1abe03ead35b58a37577a85e7ad55512d738c4bcf53596a92a5ead163a51185a"} Aug 01 00:33:44.261253 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.261200 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6sqf4" event={"ID":"8fb3d89c-6d48-445c-9726-4b33c252e881","Type":"ContainerStarted","Data":"0262b7e2e0dc077eaff5d0b939839a5de694a2af734dd2b25e9e202f4d5506b3"} Aug 01 00:33:44.265473 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.265408 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-npmz6" event={"ID":"89569cc7-3743-47cf-b288-2dac9c3e5f01","Type":"ContainerStarted","Data":"ad723d545ec8a9e76237fb4f2a20d32aa95e9fecbbad9aa56998970f6e3febb8"} Aug 01 00:33:44.267162 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.267139 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j4kdm" event={"ID":"00392c01-c744-4f1f-be3a-6eafbc64af50","Type":"ContainerStarted","Data":"0ad9611c7a7f0a0f2ecc78951adfaf6af0662735fe1f54696685b5bdb2775bdb"} Aug 01 00:33:44.269370 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.269338 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" event={"ID":"d78001921e659c8098fde0e8ca179c5e","Type":"ContainerStarted","Data":"816802dfc82fda21e6d239beef194a60a8331168497708779b9f1dd165d5c069"} Aug 01 00:33:44.272929 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.272904 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vm2k4" event={"ID":"06e31806-756d-4782-9193-ac8ce656d17b","Type":"ContainerStarted","Data":"85e461d00a93bebefb101c3c756f86b9f0a03b1d6ca004775b30e3650f0da4f7"} Aug 01 00:33:44.275833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.275801 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x4mkf" event={"ID":"10cbd086-8ace-48e5-bda2-915b5ba02111","Type":"ContainerStarted","Data":"eab03267b578581332b1f6476da0c009b1c80ae188fa350ff889703dba97965a"} Aug 01 00:33:44.277528 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.277505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"b554c01b9aecad6d72258b6928004abcbff627ccc7cf1c9a70417ff9b3c22acf"} Aug 01 00:33:44.283669 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.283645 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" event={"ID":"e7ccfef67ed5bb286014d58f5d8b58af","Type":"ContainerStarted","Data":"e53d2a2b0a4831343314e7ded32b831419ffc080e525377d9209fea103104909"} Aug 01 00:33:44.747393 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.747351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:44.747586 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.747575 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:44.747657 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.747637 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:33:46.747619202 +0000 UTC m=+4.957247651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:44.848787 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.848751 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:44.848950 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.848886 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:44.848950 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.848948 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret podName:d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f nodeName:}" failed. No retries permitted until 2026-08-01 00:33:46.848929988 +0000 UTC m=+5.058558425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret") pod "global-pull-secret-syncer-d85hp" (UID: "d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:44.949857 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:44.949819 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:44.950063 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.950023 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:33:44.950063 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.950050 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:33:44.950063 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.950065 2568 projected.go:196] Error preparing data for projected volume kube-api-access-tbtth for pod openshift-network-diagnostics/network-check-target-fgkhq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:44.950345 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:44.950125 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth podName:ef0430f8-146f-411e-9581-b62cf7e57a76 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:46.950106927 +0000 UTC m=+5.159735385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbtth" (UniqueName: "kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth") pod "network-check-target-fgkhq" (UID: "ef0430f8-146f-411e-9581-b62cf7e57a76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:45.167684 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:45.167555 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-31 00:28:43 +0000 UTC" deadline="2028-04-16 12:49:05.085937852 +0000 UTC" Aug 01 00:33:45.167684 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:45.167607 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14988h15m19.918349115s" Aug 01 00:33:45.225779 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:45.225053 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:45.225779 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:45.225188 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:45.225779 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:45.225613 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:45.225779 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:45.225711 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:33:45.922378 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:45.922146 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:33:46.224527 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:46.224458 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:46.224939 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.224610 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:46.764992 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:46.764945 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:46.765185 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.765071 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:46.765185 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.765124 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:33:50.765109178 +0000 UTC m=+8.974737616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:46.865624 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:46.865573 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:46.865807 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.865750 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:46.865876 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.865812 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret podName:d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f nodeName:}" failed. No retries permitted until 2026-08-01 00:33:50.865794138 +0000 UTC m=+9.075422588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret") pod "global-pull-secret-syncer-d85hp" (UID: "d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:46.966348 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:46.966311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:46.966570 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.966485 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:33:46.966570 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.966508 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:33:46.966570 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.966523 2568 projected.go:196] Error preparing data for projected volume kube-api-access-tbtth for pod openshift-network-diagnostics/network-check-target-fgkhq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:46.966722 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:46.966589 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth podName:ef0430f8-146f-411e-9581-b62cf7e57a76 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:50.966570317 +0000 UTC m=+9.176198765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbtth" (UniqueName: "kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth") pod "network-check-target-fgkhq" (UID: "ef0430f8-146f-411e-9581-b62cf7e57a76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:47.225046 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:47.224954 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:47.225521 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:47.225113 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:33:47.225658 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:47.225639 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:47.225801 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:47.225780 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:48.226836 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:48.226798 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:48.227309 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:48.226914 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:49.225024 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:49.224993 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:49.225185 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:49.225115 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:33:49.225185 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:49.225167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:49.225339 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:49.225296 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:50.224754 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:50.224722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:50.225104 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.224853 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:50.794956 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:50.794910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:50.795143 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.795084 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:50.795143 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.795136 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:33:58.795122864 +0000 UTC m=+17.004751299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:50.896267 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:50.896233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:50.896433 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.896414 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:50.896509 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.896487 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret podName:d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f nodeName:}" failed. No retries permitted until 2026-08-01 00:33:58.89646546 +0000 UTC m=+17.106093900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret") pod "global-pull-secret-syncer-d85hp" (UID: "d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:50.997326 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:50.997290 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:50.997513 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.997478 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:33:50.997513 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.997501 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:33:50.997513 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.997510 2568 projected.go:196] Error preparing data for projected volume kube-api-access-tbtth for pod openshift-network-diagnostics/network-check-target-fgkhq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:50.997619 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:50.997556 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth podName:ef0430f8-146f-411e-9581-b62cf7e57a76 nodeName:}" failed. No retries permitted until 2026-08-01 00:33:58.997542321 +0000 UTC m=+17.207170775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbtth" (UniqueName: "kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth") pod "network-check-target-fgkhq" (UID: "ef0430f8-146f-411e-9581-b62cf7e57a76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:51.224676 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:51.224591 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:51.224836 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:51.224591 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:51.224836 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:51.224716 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:33:51.224836 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:51.224797 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:52.225427 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:52.225257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:52.225871 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:52.225498 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:53.225711 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:53.225144 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:53.225711 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:53.225270 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:53.225711 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:53.225410 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:53.225711 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:53.225468 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:33:54.225289 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:54.225257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:54.225479 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:54.225398 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:55.225075 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:55.225044 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:55.225456 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:55.225044 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:55.225456 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:55.225146 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:55.225456 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:55.225264 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:33:56.224711 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:56.224273 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:56.224711 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:56.224396 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:57.224386 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:57.224353 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:57.224817 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:57.224480 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:57.224817 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:57.224539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:57.224817 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:57.224641 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:33:58.225037 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:58.224999 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:58.225462 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:58.225123 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:33:58.856526 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:58.856496 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:58.856724 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:58.856661 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:58.856782 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:58.856729 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:34:14.85671174 +0000 UTC m=+33.066340174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:33:58.957003 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:58.956974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:33:58.957129 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:58.957081 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:58.957129 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:58.957125 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret podName:d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f nodeName:}" failed. No retries permitted until 2026-08-01 00:34:14.957112535 +0000 UTC m=+33.166740970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret") pod "global-pull-secret-syncer-d85hp" (UID: "d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:33:59.058149 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:59.058120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:59.058333 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:59.058283 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:33:59.058333 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:59.058302 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:33:59.058446 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:59.058334 2568 projected.go:196] Error preparing data for projected volume kube-api-access-tbtth for pod openshift-network-diagnostics/network-check-target-fgkhq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:59.058446 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:59.058386 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth podName:ef0430f8-146f-411e-9581-b62cf7e57a76 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:15.058372615 +0000 UTC m=+33.268001053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbtth" (UniqueName: "kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth") pod "network-check-target-fgkhq" (UID: "ef0430f8-146f-411e-9581-b62cf7e57a76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:33:59.225139 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:59.225104 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:33:59.225139 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:33:59.225136 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:33:59.225618 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:59.225248 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:33:59.225618 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:33:59.225328 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:00.225049 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:00.225020 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:00.225267 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:00.225144 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:01.225187 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:01.225138 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:01.225399 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:01.225138 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:01.225399 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:01.225300 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:01.225399 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:01.225340 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:02.225038 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:02.225002 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:02.225202 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:02.225108 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:03.224188 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:03.224162 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:03.224537 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:03.224270 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:03.224537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:03.224321 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:03.224537 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:03.224418 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:04.224852 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.224699 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:04.225614 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:04.224928 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:04.329742 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.329503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"d2c966bd39a57d4d36480cc93271ed41817ce16b2df7dcd8c5309b07ee8d4e20"} Aug 01 00:34:04.329922 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.329769 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"208ac56c2223cf40c8237084f7d427f3f5865f307343acec37a78f88c2af7f8f"} Aug 01 00:34:04.329922 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.329817 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"be7754a59eb251ee8968cab83aad091da549f5d0934f90b980bd3feb71d13855"} Aug 01 00:34:04.329922 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.329855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"7dd2b1bc4fa6f5f54a71feea076053dbf58b806f1886f895e4a8a2e4ae4b3952"} Aug 01 00:34:04.329922 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.329883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"5ff47256e8853544b17dc83703c375b3b1ce7eec65b70808bc251e5a2e3397a5"} Aug 01 00:34:04.329922 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.329909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"137ff4ca5b0be383063a2f427965d7ebd55bef145ba775c8ccc449d23aa17b5f"} Aug 01 00:34:04.330828 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.330809 2568 generic.go:356] "Generic (PLEG): container finished" podID="e7ccfef67ed5bb286014d58f5d8b58af" containerID="b2c7b09592a5ea3483882c0ee4b06b7a37b22f6baebae61a15cb57660d8fcb34" exitCode=0 Aug 01 00:34:04.330893 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.330877 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" event={"ID":"e7ccfef67ed5bb286014d58f5d8b58af","Type":"ContainerDied","Data":"b2c7b09592a5ea3483882c0ee4b06b7a37b22f6baebae61a15cb57660d8fcb34"} Aug 01 00:34:04.332200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.332178 2568 generic.go:356] "Generic (PLEG): container finished" podID="78d38db1-2bcf-4778-8227-2956eec1f6f0" containerID="8da2ba527426f7ef9c87a6a707dfe1b91220e30ee7344890d8a3c0b51e1f8e85" exitCode=0 Aug 01 00:34:04.332318 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.332254 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerDied","Data":"8da2ba527426f7ef9c87a6a707dfe1b91220e30ee7344890d8a3c0b51e1f8e85"} Aug 01 00:34:04.333714 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.333640 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" event={"ID":"8afb46d3-d50e-4f0c-b804-1b33b989534e","Type":"ContainerStarted","Data":"0ed40f969c5d2cd5e29083bb1c96827783f8c904c1f3736d429f06b01fdb85a2"} Aug 01 00:34:04.335168 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.335136 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-75wgw" event={"ID":"fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a","Type":"ContainerStarted","Data":"b191e2a818e2a4cc9232a8aa9a8c78c4083f6d25c6560b273804bed1593564e8"} Aug 01 00:34:04.336557 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.336536 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6sqf4" event={"ID":"8fb3d89c-6d48-445c-9726-4b33c252e881","Type":"ContainerStarted","Data":"09e3da65ca7e99be13e306e06e654cfb4809f35e689c70b3906ce4ba1ec9bbfe"} Aug 01 00:34:04.338014 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.337996 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-npmz6" event={"ID":"89569cc7-3743-47cf-b288-2dac9c3e5f01","Type":"ContainerStarted","Data":"00a9b0ebdc8ac3cb9ca2048b2e06d08b78ef171d7e3efc12ed9b49e3d4946753"} Aug 01 00:34:04.339473 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.339445 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" event={"ID":"d78001921e659c8098fde0e8ca179c5e","Type":"ContainerStarted","Data":"87e4b61062f7a8657f11ab3785464b36732457a4272c00b7b9e0879be7e45ca5"} Aug 01 00:34:04.340804 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.340783 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vm2k4" event={"ID":"06e31806-756d-4782-9193-ac8ce656d17b","Type":"ContainerStarted","Data":"7b6a28610ce70425d53fc4680194112c73282fbad9e58f1b875e61e00475b2d4"} Aug 01 00:34:04.341917 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.341898 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x4mkf" event={"ID":"10cbd086-8ace-48e5-bda2-915b5ba02111","Type":"ContainerStarted","Data":"3fb835554a01a20c7431b86c51c622aa924a3a8d4532b19541489aa7a9b591fd"} Aug 01 00:34:04.385758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.385715 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x4mkf" podStartSLOduration=2.454816366 podStartE2EDuration="22.385702588s" totalImagesPullingTime="19.930886222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:04.385628737 +0000 UTC m=+22.595257194" watchObservedRunningTime="2026-08-01 00:34:04.385702588 +0000 UTC m=+22.595331045" Aug 01 00:34:04.385909 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.385883 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-42.ec2.internal" podStartSLOduration=21.385879116 podStartE2EDuration="21.385879116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:04.36744644 +0000 UTC m=+22.577074896" watchObservedRunningTime="2026-08-01 00:34:04.385879116 +0000 UTC m=+22.595507573" Aug 01 00:34:04.401167 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.401131 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6sqf4" podStartSLOduration=2.305701612 podStartE2EDuration="22.401120221s" totalImagesPullingTime="20.095418609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:04.401012701 +0000 UTC m=+22.610641170" watchObservedRunningTime="2026-08-01 00:34:04.401120221 +0000 UTC m=+22.610748678" Aug 01 00:34:04.416530 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.416488 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-npmz6" podStartSLOduration=2.4841429760000002 podStartE2EDuration="22.416478119s" totalImagesPullingTime="19.932335143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:04.416295539 +0000 UTC m=+22.625924007" watchObservedRunningTime="2026-08-01 00:34:04.416478119 +0000 UTC m=+22.626106576" Aug 01 00:34:04.429740 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:04.429701 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-75wgw" podStartSLOduration=2.5811581439999998 podStartE2EDuration="22.429691047s" totalImagesPullingTime="19.848532903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:04.42928275 +0000 UTC m=+22.638911206" watchObservedRunningTime="2026-08-01 00:34:04.429691047 +0000 UTC m=+22.639319512" Aug 01 00:34:05.224380 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:05.224284 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:05.224380 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:05.224334 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:05.224598 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:05.224425 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:05.224653 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:05.224628 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:05.345073 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:05.345042 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j4kdm" event={"ID":"00392c01-c744-4f1f-be3a-6eafbc64af50","Type":"ContainerStarted","Data":"104e5a284c0327f884138b18c3acdeba53fc4e082d5a2896182f4954f6cac499"} Aug 01 00:34:05.358370 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:05.358331 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vm2k4" podStartSLOduration=3.549116197 podStartE2EDuration="23.358314916s" totalImagesPullingTime="19.809198719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:04.455631836 +0000 UTC m=+22.665260293" watchObservedRunningTime="2026-08-01 00:34:05.358314916 +0000 UTC m=+23.567943373" Aug 01 00:34:05.370856 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:05.370822 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j4kdm" podStartSLOduration=3.461588565 podStartE2EDuration="23.370807629s" totalImagesPullingTime="19.909219064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:05.370523154 +0000 UTC m=+23.580151615" watchObservedRunningTime="2026-08-01 00:34:05.370807629 +0000 UTC m=+23.580436084" Aug 01 00:34:05.505480 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:05.505451 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 01 00:34:06.199041 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:06.198945 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-01T00:34:05.505474292Z","UUID":"774f1e17-0704-43d6-8a40-5854b37c93ce","Handler":null,"Name":"","Endpoint":""} Aug 01 00:34:06.200382 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:06.200364 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 01 00:34:06.200522 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:06.200406 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 01 00:34:06.224764 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:06.224738 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:06.224912 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:06.224848 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:06.349816 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:06.349780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" event={"ID":"e7ccfef67ed5bb286014d58f5d8b58af","Type":"ContainerStarted","Data":"a3454aaa4641d0ab987d63967cc3c7e244953820593037112ca2b15f8216f135"} Aug 01 00:34:06.351991 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:06.351963 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" event={"ID":"8afb46d3-d50e-4f0c-b804-1b33b989534e","Type":"ContainerStarted","Data":"dcfddde5a64bd274404a7d0f0693cbcab3d4626271608361f2ecd612414fab13"} Aug 01 00:34:06.368324 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:06.368272 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-42.ec2.internal" podStartSLOduration=23.368252816 podStartE2EDuration="23.368252816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:06.367373744 +0000 UTC m=+24.577002202" watchObservedRunningTime="2026-08-01 00:34:06.368252816 +0000 UTC m=+24.577881274" Aug 01 00:34:07.224421 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:07.224341 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:07.224570 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:07.224464 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:07.224570 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:07.224533 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:07.224682 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:07.224634 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:07.357377 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:07.357339 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" event={"ID":"8afb46d3-d50e-4f0c-b804-1b33b989534e","Type":"ContainerStarted","Data":"9e859278e6117bf2cb5c7d4ad4fd178da75085666c465b4f92934461837a95cf"} Aug 01 00:34:07.361229 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:07.361184 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"a2974c3549538c3c9cdbcd84c34ee292c58a25484f9bed37c05992b19ad6b583"} Aug 01 00:34:07.369446 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:07.369421 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:34:07.370063 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:07.370044 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:34:07.374475 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:07.374433 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf7nh" podStartSLOduration=2.833570207 podStartE2EDuration="25.374420339s" totalImagesPullingTime="22.540850132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:07.374334282 +0000 UTC m=+25.583962720" watchObservedRunningTime="2026-08-01 00:34:07.374420339 +0000 UTC m=+25.584048795" Aug 01 00:34:08.224480 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:08.224451 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:08.224666 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:08.224579 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:08.364652 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:08.364097 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:34:08.364652 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:08.364603 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-75wgw" Aug 01 00:34:09.224365 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:09.224328 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:09.224540 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:09.224375 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:09.224540 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:09.224461 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:09.224658 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:09.224564 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:10.224918 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:10.224890 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:10.225555 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:10.224992 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:10.372152 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:10.372118 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" event={"ID":"0007ab3a-bc9a-4a12-9a80-baee7909713c","Type":"ContainerStarted","Data":"c8c0e06c440ba1d058ce58f649c0717af5650dd72a046a535a2171f5a4587441"} Aug 01 00:34:10.372518 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:10.372498 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:34:10.374040 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:10.374011 2568 generic.go:356] "Generic (PLEG): container finished" podID="78d38db1-2bcf-4778-8227-2956eec1f6f0" containerID="b8e9ec4fa2a09527c65138f4c824e8c5732328637c16fc4d1df44d050dbb19dc" exitCode=0 Aug 01 00:34:10.374160 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:10.374086 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerDied","Data":"b8e9ec4fa2a09527c65138f4c824e8c5732328637c16fc4d1df44d050dbb19dc"} Aug 01 00:34:10.388703 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:10.388684 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:34:10.396049 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:10.396008 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" podStartSLOduration=8.408977308 podStartE2EDuration="28.395995146s" totalImagesPullingTime="19.987017838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:10.394909322 +0000 UTC m=+28.604537781" watchObservedRunningTime="2026-08-01 00:34:10.395995146 +0000 UTC m=+28.605623602" Aug 01 00:34:11.224857 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.224564 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:11.225026 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:11.224934 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:11.225026 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.225010 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:11.225416 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:11.225080 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:11.379009 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.378973 2568 generic.go:356] "Generic (PLEG): container finished" podID="78d38db1-2bcf-4778-8227-2956eec1f6f0" containerID="30aaf243c8d0728202f5ffbff9a3b734cadf228dd0373141f3afb06008988960" exitCode=0 Aug 01 00:34:11.379351 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.379058 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerDied","Data":"30aaf243c8d0728202f5ffbff9a3b734cadf228dd0373141f3afb06008988960"} Aug 01 00:34:11.379558 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.379537 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:34:11.379671 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.379605 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:34:11.395016 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.394995 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:34:11.888481 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.888446 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d85hp"] Aug 01 00:34:11.888662 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.888569 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:11.888732 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:11.888656 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:11.893484 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.893458 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-249m9"] Aug 01 00:34:11.893607 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.893514 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fgkhq"] Aug 01 00:34:11.893607 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.893583 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:11.893716 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:11.893657 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:11.893776 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:11.893727 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:11.893864 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:11.893797 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:12.383310 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:12.383271 2568 generic.go:356] "Generic (PLEG): container finished" podID="78d38db1-2bcf-4778-8227-2956eec1f6f0" containerID="321ad4df9f158e3693abc1f494c8fadef06b212ee1768560f71c0348a69ba87c" exitCode=0 Aug 01 00:34:12.383753 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:12.383361 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerDied","Data":"321ad4df9f158e3693abc1f494c8fadef06b212ee1768560f71c0348a69ba87c"} Aug 01 00:34:13.224286 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:13.224248 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:13.224448 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:13.224373 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:14.228445 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:14.228368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:14.228445 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:14.228412 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:14.228906 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:14.228511 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:14.228906 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:14.228622 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:14.875436 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:14.875400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:14.875597 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:14.875550 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:34:14.875638 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:14.875622 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:34:46.875600352 +0000 UTC m=+65.085228802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:34:14.976112 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:14.976033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:14.976263 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:14.976148 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:34:14.976263 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:14.976200 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret podName:d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f nodeName:}" failed. No retries permitted until 2026-08-01 00:34:46.976185997 +0000 UTC m=+65.185814437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret") pod "global-pull-secret-syncer-d85hp" (UID: "d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:34:15.077205 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:15.077166 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:15.077436 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:15.077339 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:34:15.077436 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:15.077368 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:34:15.077436 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:15.077382 2568 projected.go:196] Error preparing data for projected volume kube-api-access-tbtth for pod openshift-network-diagnostics/network-check-target-fgkhq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:34:15.077565 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:15.077449 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth podName:ef0430f8-146f-411e-9581-b62cf7e57a76 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:47.077433688 +0000 UTC m=+65.287062123 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbtth" (UniqueName: "kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth") pod "network-check-target-fgkhq" (UID: "ef0430f8-146f-411e-9581-b62cf7e57a76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:34:15.224916 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:15.224884 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:15.225075 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:15.225004 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d85hp" podUID="d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f" Aug 01 00:34:16.225073 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.225036 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:16.225486 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:16.225162 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgkhq" podUID="ef0430f8-146f-411e-9581-b62cf7e57a76" Aug 01 00:34:16.225486 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.225275 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:16.225486 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:16.225402 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:34:16.636763 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.636738 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-42.ec2.internal" event="NodeReady" Aug 01 00:34:16.637035 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.636894 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 01 00:34:16.675182 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.675150 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67f6648b84-t5vxk"] Aug 01 00:34:16.702969 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.702922 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kvg6c"] Aug 01 00:34:16.703160 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.703019 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.705572 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.705552 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4zgh7\"" Aug 01 00:34:16.707240 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.706250 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 01 00:34:16.707240 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.706475 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 01 00:34:16.707619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.707600 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 01 00:34:16.711541 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.711521 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 01 00:34:16.723102 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.723083 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67f6648b84-t5vxk"] Aug 01 00:34:16.723202 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.723120 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vpbm6"] Aug 01 00:34:16.723265 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.723206 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kvg6c" Aug 01 00:34:16.725595 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.725576 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 01 00:34:16.725704 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.725581 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-msh2l\"" Aug 01 00:34:16.725704 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.725581 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 01 00:34:16.725813 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.725774 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 01 00:34:16.744652 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.744631 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kvg6c"] Aug 01 00:34:16.744774 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.744664 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpbm6"] Aug 01 00:34:16.744774 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.744761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.746961 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.746939 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 01 00:34:16.747078 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.746945 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 01 00:34:16.747078 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.747001 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qd5dz\"" Aug 01 00:34:16.790707 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.790676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-certificates\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.790853 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.790715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-bound-sa-token\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.790853 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.790828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-trusted-ca\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.790943 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.790881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxrx\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-kube-api-access-jmxrx\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.790943 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.790936 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-image-registry-private-configuration\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.791043 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.790956 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.791043 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.790976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59dd5e67-b383-49a7-bae0-17338c424d9b-ca-trust-extracted\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.791043 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.790996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-installation-pull-secrets\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.847410 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.847373 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:16.892367 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-image-registry-private-configuration\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.892367 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892335 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw62d\" (UniqueName: \"kubernetes.io/projected/8cdf8609-6802-4dc9-b5b6-ca974534987b-kube-api-access-mw62d\") pod \"ingress-canary-kvg6c\" (UID: \"8cdf8609-6802-4dc9-b5b6-ca974534987b\") " pod="openshift-ingress-canary/ingress-canary-kvg6c" Aug 01 00:34:16.892367 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.892603 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892372 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59dd5e67-b383-49a7-bae0-17338c424d9b-ca-trust-extracted\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.892603 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-installation-pull-secrets\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.892603 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892459 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.892603 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:16.892461 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:34:16.892603 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892495 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30615951-9c11-4b6b-9b0a-00813d2ec60e-tmp-dir\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.892603 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:16.892507 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:34:16.892603 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892521 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjkv\" (UniqueName: \"kubernetes.io/projected/30615951-9c11-4b6b-9b0a-00813d2ec60e-kube-api-access-kpjkv\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.892603 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:16.892564 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:34:17.392544891 +0000 UTC m=+35.602173343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:34:16.892937 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-certificates\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.893004 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-bound-sa-token\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.893071 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.893057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cdf8609-6802-4dc9-b5b6-ca974534987b-cert\") pod \"ingress-canary-kvg6c\" (UID: \"8cdf8609-6802-4dc9-b5b6-ca974534987b\") " pod="openshift-ingress-canary/ingress-canary-kvg6c" Aug 01 00:34:16.893126 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.893113 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30615951-9c11-4b6b-9b0a-00813d2ec60e-config-volume\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.893175 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.892924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59dd5e67-b383-49a7-bae0-17338c424d9b-ca-trust-extracted\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.893526 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.893503 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-certificates\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.893658 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.893642 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-trusted-ca\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.893723 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.893700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxrx\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-kube-api-access-jmxrx\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.897546 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.897517 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-installation-pull-secrets\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.897678 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.897661 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-trusted-ca\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.898331 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.898308 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-image-registry-private-configuration\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.904422 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.904301 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-bound-sa-token\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.904580 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.904560 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxrx\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-kube-api-access-jmxrx\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:16.942509 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.942490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:16.994493 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.994472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cdf8609-6802-4dc9-b5b6-ca974534987b-cert\") pod \"ingress-canary-kvg6c\" (UID: \"8cdf8609-6802-4dc9-b5b6-ca974534987b\") " pod="openshift-ingress-canary/ingress-canary-kvg6c" Aug 01 00:34:16.994603 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.994511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30615951-9c11-4b6b-9b0a-00813d2ec60e-config-volume\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.994603 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.994541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw62d\" (UniqueName: \"kubernetes.io/projected/8cdf8609-6802-4dc9-b5b6-ca974534987b-kube-api-access-mw62d\") pod \"ingress-canary-kvg6c\" (UID: \"8cdf8609-6802-4dc9-b5b6-ca974534987b\") " pod="openshift-ingress-canary/ingress-canary-kvg6c" Aug 01 00:34:16.994717 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.994694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.994773 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.994744 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30615951-9c11-4b6b-9b0a-00813d2ec60e-tmp-dir\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.994829 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.994773 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjkv\" (UniqueName: \"kubernetes.io/projected/30615951-9c11-4b6b-9b0a-00813d2ec60e-kube-api-access-kpjkv\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.994874 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:16.994850 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:34:16.994918 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:16.994911 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:34:17.494895773 +0000 UTC m=+35.704524208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:34:16.995020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.995001 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30615951-9c11-4b6b-9b0a-00813d2ec60e-tmp-dir\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:16.995061 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:16.995017 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30615951-9c11-4b6b-9b0a-00813d2ec60e-config-volume\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:17.000338 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.000319 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cdf8609-6802-4dc9-b5b6-ca974534987b-cert\") pod \"ingress-canary-kvg6c\" (UID: \"8cdf8609-6802-4dc9-b5b6-ca974534987b\") " pod="openshift-ingress-canary/ingress-canary-kvg6c" Aug 01 00:34:17.009778 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.009757 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw62d\" (UniqueName: \"kubernetes.io/projected/8cdf8609-6802-4dc9-b5b6-ca974534987b-kube-api-access-mw62d\") pod \"ingress-canary-kvg6c\" (UID: \"8cdf8609-6802-4dc9-b5b6-ca974534987b\") " pod="openshift-ingress-canary/ingress-canary-kvg6c" Aug 01 00:34:17.010073 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.010054 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjkv\" (UniqueName: \"kubernetes.io/projected/30615951-9c11-4b6b-9b0a-00813d2ec60e-kube-api-access-kpjkv\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:17.032117 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.032094 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kvg6c" Aug 01 00:34:17.207503 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.207472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kvg6c"] Aug 01 00:34:17.222355 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:34:17.222329 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdf8609_6802_4dc9_b5b6_ca974534987b.slice/crio-0a93eece4abd40cfcdc7b0143aee54c6c1a5d2643b25ad0c82da3e560bcd7cc2 WatchSource:0}: Error finding container 0a93eece4abd40cfcdc7b0143aee54c6c1a5d2643b25ad0c82da3e560bcd7cc2: Status 404 returned error can't find the container with id 0a93eece4abd40cfcdc7b0143aee54c6c1a5d2643b25ad0c82da3e560bcd7cc2 Aug 01 00:34:17.225018 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.225003 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:17.227744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.227726 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 01 00:34:17.395578 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.395546 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kvg6c" event={"ID":"8cdf8609-6802-4dc9-b5b6-ca974534987b","Type":"ContainerStarted","Data":"0a93eece4abd40cfcdc7b0143aee54c6c1a5d2643b25ad0c82da3e560bcd7cc2"} Aug 01 00:34:17.396374 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.396355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:17.396529 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:17.396515 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:34:17.396580 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:17.396532 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:34:17.396630 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:17.396585 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:34:18.396561771 +0000 UTC m=+36.606190218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:34:17.398875 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.398842 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerStarted","Data":"fffae12e25e3c1b7305f004db2c3e48e86d9b3dea4b71b22c304090a88cbbfb0"} Aug 01 00:34:17.497429 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.497406 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:17.497537 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:17.497525 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:34:17.497586 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:17.497578 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:34:18.497564948 +0000 UTC m=+36.707193386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:34:17.526164 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.526136 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:17.587526 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:17.587494 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:18.224782 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.224747 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:18.225038 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.224949 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:18.227173 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.227150 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 01 00:34:18.227292 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.227156 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 01 00:34:18.227914 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.227894 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 01 00:34:18.228418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.227897 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hk5qd\"" Aug 01 00:34:18.228418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.227899 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hdfd9\"" Aug 01 00:34:18.403932 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.403904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:18.404113 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:18.404057 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:34:18.404113 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:18.404072 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:34:18.404267 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:18.404130 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:34:20.404109138 +0000 UTC m=+38.613737576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:34:18.404768 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.404741 2568 generic.go:356] "Generic (PLEG): container finished" podID="78d38db1-2bcf-4778-8227-2956eec1f6f0" containerID="fffae12e25e3c1b7305f004db2c3e48e86d9b3dea4b71b22c304090a88cbbfb0" exitCode=0 Aug 01 00:34:18.404873 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.404779 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerDied","Data":"fffae12e25e3c1b7305f004db2c3e48e86d9b3dea4b71b22c304090a88cbbfb0"} Aug 01 00:34:18.505059 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:18.504617 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:18.505059 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:18.504757 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:34:18.505059 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:18.504802 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:34:20.504789015 +0000 UTC m=+38.714417450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:34:19.411194 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:19.411160 2568 generic.go:356] "Generic (PLEG): container finished" podID="78d38db1-2bcf-4778-8227-2956eec1f6f0" containerID="da3a912937024c052527c24c6b71664bf43f1137fd5a542ba4560819c9d5d662" exitCode=0 Aug 01 00:34:19.411808 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:19.411243 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerDied","Data":"da3a912937024c052527c24c6b71664bf43f1137fd5a542ba4560819c9d5d662"} Aug 01 00:34:19.469892 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:19.469860 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:20.414976 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:20.414721 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kvg6c" event={"ID":"8cdf8609-6802-4dc9-b5b6-ca974534987b","Type":"ContainerStarted","Data":"73b277a31ba0dbd55bb6f1a2fdb867c1a9e7601e12150af5f91ce812c42c7566"} Aug 01 00:34:20.417297 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:20.417266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:20.417466 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:20.417451 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:34:20.417524 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:20.417468 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:34:20.417566 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:20.417523 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:34:24.417503467 +0000 UTC m=+42.627131920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:34:20.418174 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:20.418151 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4khxh" event={"ID":"78d38db1-2bcf-4778-8227-2956eec1f6f0","Type":"ContainerStarted","Data":"22bbe248ea6c7c4f5cc998c80b219fd9e586d6c6c9e923d363fc7e495724c250"} Aug 01 00:34:20.428648 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:20.428603 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kvg6c" podStartSLOduration=1.6256081660000001 podStartE2EDuration="4.428587172s" totalImagesPullingTime="2.802979006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:20.427594379 +0000 UTC m=+38.637222835" watchObservedRunningTime="2026-08-01 00:34:20.428587172 +0000 UTC m=+38.638215624" Aug 01 00:34:20.448406 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:20.448364 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4khxh" podStartSLOduration=6.514902617 podStartE2EDuration="38.448349049s" totalImagesPullingTime="31.933446432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:20.44801759 +0000 UTC m=+38.657646039" watchObservedRunningTime="2026-08-01 00:34:20.448349049 +0000 UTC m=+38.657977507" Aug 01 00:34:20.517884 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:20.517849 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:20.518055 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:20.517970 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:34:20.518055 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:20.518033 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:34:24.518016625 +0000 UTC m=+42.727645064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:34:20.571301 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:20.571267 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:20.670693 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:20.670625 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:21.839031 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:21.839002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:21.940396 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:21.940360 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:22.040905 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:22.040870 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:24.448418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:24.448383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:24.448790 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:24.448507 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:34:24.448790 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:24.448519 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:34:24.448790 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:24.448572 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:34:32.44855387 +0000 UTC m=+50.658182322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:34:24.549041 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:24.549007 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:24.549188 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:24.549140 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:34:24.549247 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:24.549190 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:34:32.549177061 +0000 UTC m=+50.758805498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:34:32.497410 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:32.497379 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:32.497757 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:32.497492 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:34:32.497757 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:32.497505 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:34:32.497757 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:32.497560 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:34:48.497543754 +0000 UTC m=+66.707172209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:34:32.597684 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:32.597641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:32.597836 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:32.597792 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:34:32.597900 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:32.597851 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:34:48.597836876 +0000 UTC m=+66.807465311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:34:43.396316 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:43.396284 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptg24" Aug 01 00:34:46.906663 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:46.906618 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:34:46.908863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:46.908843 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 01 00:34:46.917465 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:46.917441 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 01 00:34:46.917564 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:46.917532 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:35:50.917509522 +0000 UTC m=+129.127137974 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : secret "metrics-daemon-secret" not found Aug 01 00:34:47.007415 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.007373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:47.009543 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.009527 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 01 00:34:47.021920 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.021888 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f-original-pull-secret\") pod \"global-pull-secret-syncer-d85hp\" (UID: \"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f\") " pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:47.108270 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.108232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:47.110586 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.110564 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 01 00:34:47.121264 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.121238 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 01 00:34:47.132406 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.132373 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbtth\" (UniqueName: \"kubernetes.io/projected/ef0430f8-146f-411e-9581-b62cf7e57a76-kube-api-access-tbtth\") pod \"network-check-target-fgkhq\" (UID: \"ef0430f8-146f-411e-9581-b62cf7e57a76\") " pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:47.233762 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.233722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d85hp" Aug 01 00:34:47.337383 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.337354 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hk5qd\"" Aug 01 00:34:47.346150 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.346114 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:47.362028 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.362001 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d85hp"] Aug 01 00:34:47.365367 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:34:47.365341 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e36be9_8c7d_4886_b9d1_c3e2aaf5ed5f.slice/crio-2c27b0b593cbc5904407d73ef78471b14be3cf3ad71d803bd982cf35eff67063 WatchSource:0}: Error finding container 2c27b0b593cbc5904407d73ef78471b14be3cf3ad71d803bd982cf35eff67063: Status 404 returned error can't find the container with id 2c27b0b593cbc5904407d73ef78471b14be3cf3ad71d803bd982cf35eff67063 Aug 01 00:34:47.484130 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.484050 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d85hp" event={"ID":"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f","Type":"ContainerStarted","Data":"2c27b0b593cbc5904407d73ef78471b14be3cf3ad71d803bd982cf35eff67063"} Aug 01 00:34:47.485376 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:47.485353 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fgkhq"] Aug 01 00:34:47.488924 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:34:47.488901 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0430f8_146f_411e_9581_b62cf7e57a76.slice/crio-05c2ee78a2eb68ef2cb3ac03285c1369797df2c39c4df132137b313cf60b3221 WatchSource:0}: Error finding container 05c2ee78a2eb68ef2cb3ac03285c1369797df2c39c4df132137b313cf60b3221: Status 404 returned error can't find the container with id 05c2ee78a2eb68ef2cb3ac03285c1369797df2c39c4df132137b313cf60b3221 Aug 01 00:34:48.489373 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:48.489243 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fgkhq" event={"ID":"ef0430f8-146f-411e-9581-b62cf7e57a76","Type":"ContainerStarted","Data":"05c2ee78a2eb68ef2cb3ac03285c1369797df2c39c4df132137b313cf60b3221"} Aug 01 00:34:48.519010 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:48.518970 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:34:48.519179 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:48.519107 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:34:48.519179 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:48.519123 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:34:48.519282 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:48.519189 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:35:20.519168912 +0000 UTC m=+98.728797360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:34:48.619733 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:48.619700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:34:48.619925 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:48.619889 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:34:48.619983 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:34:48.619970 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:35:20.619947735 +0000 UTC m=+98.829576193 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:34:52.500882 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:52.500826 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d85hp" event={"ID":"d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f","Type":"ContainerStarted","Data":"7290df0229500d4b57ce13614476c53aa307e1786c86eea0a266ab0cd0ea5399"} Aug 01 00:34:52.502253 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:52.502232 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fgkhq" event={"ID":"ef0430f8-146f-411e-9581-b62cf7e57a76","Type":"ContainerStarted","Data":"cb6b6cf7304e379523c4c1ee0dcbbe043837ad9aa05b5e5ce1a667ef22f5027c"} Aug 01 00:34:52.502365 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:52.502354 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:34:52.514648 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:52.514603 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d85hp" podStartSLOduration=65.982241892 podStartE2EDuration="1m10.514591881s" totalImagesPullingTime="4.532349989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:52.51367395 +0000 UTC m=+70.723302407" watchObservedRunningTime="2026-08-01 00:34:52.514591881 +0000 UTC m=+70.724220338" Aug 01 00:34:52.526928 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:34:52.526887 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fgkhq" podStartSLOduration=66.122278404 podStartE2EDuration="1m10.526872402s" totalImagesPullingTime="4.404593998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:52.525848723 +0000 UTC m=+70.735477192" watchObservedRunningTime="2026-08-01 00:34:52.526872402 +0000 UTC m=+70.736500860" Aug 01 00:35:20.538948 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:35:20.538901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:35:20.539433 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:35:20.539064 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:35:20.539433 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:35:20.539083 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:35:20.539433 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:35:20.539150 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:36:24.539133456 +0000 UTC m=+162.748761891 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:35:20.639351 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:35:20.639315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:35:20.639508 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:35:20.639422 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:35:20.639508 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:35:20.639479 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:36:24.639466129 +0000 UTC m=+162.849094564 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:35:23.506636 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:35:23.506602 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fgkhq" Aug 01 00:35:50.957205 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:35:50.957144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:35:50.957707 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:35:50.957330 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 01 00:35:50.957707 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:35:50.957402 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:37:52.95738606 +0000 UTC m=+251.167014495 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : secret "metrics-daemon-secret" not found Aug 01 00:36:19.715537 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:36:19.715499 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" podUID="59dd5e67-b383-49a7-bae0-17338c424d9b" Aug 01 00:36:19.754949 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:36:19.754915 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vpbm6" podUID="30615951-9c11-4b6b-9b0a-00813d2ec60e" Aug 01 00:36:20.709119 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:36:20.706370 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:36:21.239011 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:36:21.238976 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:36:24.587662 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:36:24.587606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:36:24.588049 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:36:24.587767 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 01 00:36:24.588049 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:36:24.587790 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67f6648b84-t5vxk: secret "image-registry-tls" not found Aug 01 00:36:24.588049 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:36:24.587842 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls podName:59dd5e67-b383-49a7-bae0-17338c424d9b nodeName:}" failed. No retries permitted until 2026-08-01 00:38:26.587826675 +0000 UTC m=+284.797455110 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls") pod "image-registry-67f6648b84-t5vxk" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b") : secret "image-registry-tls" not found Aug 01 00:36:24.688331 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:36:24.688279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:36:24.688520 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:36:24.688433 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:36:24.688520 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:36:24.688493 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls podName:30615951-9c11-4b6b-9b0a-00813d2ec60e nodeName:}" failed. No retries permitted until 2026-08-01 00:38:26.688477533 +0000 UTC m=+284.898105968 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls") pod "dns-default-vpbm6" (UID: "30615951-9c11-4b6b-9b0a-00813d2ec60e") : secret "dns-default-metrics-tls" not found Aug 01 00:36:32.226203 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:36:32.226169 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpbm6" Aug 01 00:36:33.224393 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:36:33.224289 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:37:38.701792 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.701757 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5"] Aug 01 00:37:38.704622 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.704602 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5" Aug 01 00:37:38.706683 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.706656 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9sf8v\"" Aug 01 00:37:38.709799 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.709772 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg"] Aug 01 00:37:38.712839 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.712815 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh"] Aug 01 00:37:38.712946 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.712928 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg" Aug 01 00:37:38.715475 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.715451 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nnd59\"" Aug 01 00:37:38.715587 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.715459 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 01 00:37:38.715672 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.715652 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64956b7558-rqmzr"] Aug 01 00:37:38.715794 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.715779 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:38.717429 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.717408 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 01 00:37:38.717769 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.717751 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j5cqs\"" Aug 01 00:37:38.718084 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.718069 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 01 00:37:38.718586 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.718565 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:38.719067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.719044 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:37:38.719187 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.719138 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 01 00:37:38.719421 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.719404 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 01 00:37:38.720996 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.720977 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 01 00:37:38.720996 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.720992 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 01 00:37:38.721169 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.720983 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5"] Aug 01 00:37:38.721169 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.721075 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 01 00:37:38.721169 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.721115 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 01 00:37:38.721379 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.721316 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 01 00:37:38.721473 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.721315 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 01 00:37:38.721554 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.721321 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rr59w\"" Aug 01 00:37:38.723763 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.723743 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg"] Aug 01 00:37:38.724805 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.724787 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh"] Aug 01 00:37:38.735351 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.735322 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64956b7558-rqmzr"] Aug 01 00:37:38.807662 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.807628 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz"] Aug 01 00:37:38.810462 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.810436 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj"] Aug 01 00:37:38.810665 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.810642 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:38.810775 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.810751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv242\" (UniqueName: \"kubernetes.io/projected/143164f9-1292-4b96-9f5f-1b3958ae6f8d-kube-api-access-pv242\") pod \"network-check-source-65bdbbff45-jfpj5\" (UID: \"143164f9-1292-4b96-9f5f-1b3958ae6f8d\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5" Aug 01 00:37:38.810857 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.810836 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpplz\" (UniqueName: \"kubernetes.io/projected/976974b1-114c-4c8e-96ce-86a0a2fb7552-kube-api-access-rpplz\") pod \"volume-data-source-validator-5b8776669-jrrzg\" (UID: \"976974b1-114c-4c8e-96ce-86a0a2fb7552\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg" Aug 01 00:37:38.812833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.812813 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 01 00:37:38.813034 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.813019 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 01 00:37:38.813119 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.813056 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-b8jrr\"" Aug 01 00:37:38.813119 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.813111 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 01 00:37:38.813352 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.813337 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:37:38.813753 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.813735 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-zmdst"] Aug 01 00:37:38.813888 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.813870 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:38.817798 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.817772 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-dx56d"] Aug 01 00:37:38.818275 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.818254 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:38.818691 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.818669 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 01 00:37:38.818789 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.818675 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:37:38.818943 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.818913 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7n46z\"" Aug 01 00:37:38.819378 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.819362 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 01 00:37:38.820971 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.820952 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 01 00:37:38.821106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.821004 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 01 00:37:38.821174 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.821031 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 01 00:37:38.821250 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.821042 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 01 00:37:38.821839 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.821819 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f5hgl\"" Aug 01 00:37:38.822111 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.822086 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz"] Aug 01 00:37:38.822230 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.822183 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:38.823405 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.823299 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj"] Aug 01 00:37:38.827409 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.827391 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-zmdst"] Aug 01 00:37:38.827904 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.827885 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 01 00:37:38.827991 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.827929 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 01 00:37:38.827991 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.827951 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-44cbq\"" Aug 01 00:37:38.829389 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.829368 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 01 00:37:38.843755 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.843429 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-dx56d"] Aug 01 00:37:38.903019 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.902978 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p"] Aug 01 00:37:38.906070 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.906048 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:38.908013 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.907981 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 01 00:37:38.908013 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.907989 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 01 00:37:38.908249 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.908129 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:37:38.908333 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.908320 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 01 00:37:38.908371 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.908338 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-l8swb\"" Aug 01 00:37:38.911309 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911290 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpplz\" (UniqueName: \"kubernetes.io/projected/976974b1-114c-4c8e-96ce-86a0a2fb7552-kube-api-access-rpplz\") pod \"volume-data-source-validator-5b8776669-jrrzg\" (UID: \"976974b1-114c-4c8e-96ce-86a0a2fb7552\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg" Aug 01 00:37:38.911404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-stats-auth\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:38.911404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv242\" (UniqueName: \"kubernetes.io/projected/143164f9-1292-4b96-9f5f-1b3958ae6f8d-kube-api-access-pv242\") pod \"network-check-source-65bdbbff45-jfpj5\" (UID: \"143164f9-1292-4b96-9f5f-1b3958ae6f8d\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5" Aug 01 00:37:38.911404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-default-certificate\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:38.911553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:38.911553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911482 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzc6\" (UniqueName: \"kubernetes.io/projected/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-kube-api-access-5tzc6\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:38.911553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cac70d06-1990-4dbf-83ee-f221288845aa-config\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:38.911806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911565 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdvv\" (UniqueName: \"kubernetes.io/projected/b1fac243-00b1-4b7d-a1c9-5ba809a41789-kube-api-access-wzdvv\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:38.911806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911586 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:38.911806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cac70d06-1990-4dbf-83ee-f221288845aa-serving-cert\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:38.911806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911630 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1fac243-00b1-4b7d-a1c9-5ba809a41789-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:38.911806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:38.911806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.911677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ft7h\" (UniqueName: \"kubernetes.io/projected/cac70d06-1990-4dbf-83ee-f221288845aa-kube-api-access-9ft7h\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:38.916008 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.915987 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p"] Aug 01 00:37:38.921599 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.921577 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv242\" (UniqueName: \"kubernetes.io/projected/143164f9-1292-4b96-9f5f-1b3958ae6f8d-kube-api-access-pv242\") pod \"network-check-source-65bdbbff45-jfpj5\" (UID: \"143164f9-1292-4b96-9f5f-1b3958ae6f8d\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5" Aug 01 00:37:38.924519 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:38.924495 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpplz\" (UniqueName: \"kubernetes.io/projected/976974b1-114c-4c8e-96ce-86a0a2fb7552-kube-api-access-rpplz\") pod \"volume-data-source-validator-5b8776669-jrrzg\" (UID: \"976974b1-114c-4c8e-96ce-86a0a2fb7552\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg" Aug 01 00:37:39.012304 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012198 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e2afe34-c831-4376-a249-119e5eb68763-serving-cert\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.012304 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.012304 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-default-certificate\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.012304 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012305 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e2afe34-c831-4376-a249-119e5eb68763-trusted-ca-bundle\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzc6\" (UniqueName: \"kubernetes.io/projected/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-kube-api-access-5tzc6\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cac70d06-1990-4dbf-83ee-f221288845aa-config\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012403 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdvv\" (UniqueName: \"kubernetes.io/projected/b1fac243-00b1-4b7d-a1c9-5ba809a41789-kube-api-access-wzdvv\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.012444 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e2afe34-c831-4376-a249-119e5eb68763-tmp\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.012523 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:39.512501343 +0000 UTC m=+237.722129778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : secret "router-metrics-certs-default" not found Aug 01 00:37:39.012619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cac70d06-1990-4dbf-83ee-f221288845aa-serving-cert\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e2afe34-c831-4376-a249-119e5eb68763-snapshots\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.012715 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:39.512698095 +0000 UTC m=+237.722326549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : configmap references non-existent config key: service-ca.crt Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1fac243-00b1-4b7d-a1c9-5ba809a41789-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012794 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jts\" (UniqueName: \"kubernetes.io/projected/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-kube-api-access-d6jts\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012825 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxjrz\" (UniqueName: \"kubernetes.io/projected/7e2afe34-c831-4376-a249-119e5eb68763-kube-api-access-pxjrz\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012853 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012881 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ft7h\" (UniqueName: \"kubernetes.io/projected/cac70d06-1990-4dbf-83ee-f221288845aa-kube-api-access-9ft7h\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012957 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.012996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bl22\" (UniqueName: \"kubernetes.io/projected/d79164a3-2f9d-49b2-a10a-baca631e83a2-kube-api-access-6bl22\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.013026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e2afe34-c831-4376-a249-119e5eb68763-service-ca-bundle\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.013029 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cac70d06-1990-4dbf-83ee-f221288845aa-config\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:39.013081 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.013068 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-stats-auth\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.013905 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.013180 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:39.013905 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.013284 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls podName:b1fac243-00b1-4b7d-a1c9-5ba809a41789 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:39.513263406 +0000 UTC m=+237.722891844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-f7lwh" (UID: "b1fac243-00b1-4b7d-a1c9-5ba809a41789") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:39.013905 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.013308 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca0bbe23-8ace-42cd-8357-552e69aa9648-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:39.013905 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.013390 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:39.013905 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.013488 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b1fac243-00b1-4b7d-a1c9-5ba809a41789-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:39.014298 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.014280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5" Aug 01 00:37:39.015095 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.015074 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-default-certificate\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.015175 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.015101 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cac70d06-1990-4dbf-83ee-f221288845aa-serving-cert\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:39.015572 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.015552 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-stats-auth\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.023363 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.023306 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ft7h\" (UniqueName: \"kubernetes.io/projected/cac70d06-1990-4dbf-83ee-f221288845aa-kube-api-access-9ft7h\") pod \"service-ca-operator-784f97bcf6-8m8rz\" (UID: \"cac70d06-1990-4dbf-83ee-f221288845aa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:39.023483 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.023406 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdvv\" (UniqueName: \"kubernetes.io/projected/b1fac243-00b1-4b7d-a1c9-5ba809a41789-kube-api-access-wzdvv\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:39.023791 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.023772 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzc6\" (UniqueName: \"kubernetes.io/projected/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-kube-api-access-5tzc6\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.024304 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.024288 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg" Aug 01 00:37:39.114029 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca0bbe23-8ace-42cd-8357-552e69aa9648-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:39.114162 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114047 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:39.114162 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e2afe34-c831-4376-a249-119e5eb68763-serving-cert\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.114162 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.114162 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e2afe34-c831-4376-a249-119e5eb68763-trusted-ca-bundle\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.114162 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.114338 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.114171 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:37:39.114338 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.114254 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls podName:d79164a3-2f9d-49b2-a10a-baca631e83a2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:39.614235185 +0000 UTC m=+237.823863643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-m6qzj" (UID: "d79164a3-2f9d-49b2-a10a-baca631e83a2") : secret "samples-operator-tls" not found Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e2afe34-c831-4376-a249-119e5eb68763-tmp\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e2afe34-c831-4376-a249-119e5eb68763-snapshots\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114808 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jts\" (UniqueName: \"kubernetes.io/projected/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-kube-api-access-d6jts\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114813 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca0bbe23-8ace-42cd-8357-552e69aa9648-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxjrz\" (UniqueName: \"kubernetes.io/projected/7e2afe34-c831-4376-a249-119e5eb68763-kube-api-access-pxjrz\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114932 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bl22\" (UniqueName: \"kubernetes.io/projected/d79164a3-2f9d-49b2-a10a-baca631e83a2-kube-api-access-6bl22\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.114963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e2afe34-c831-4376-a249-119e5eb68763-service-ca-bundle\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.115056 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.115341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e2afe34-c831-4376-a249-119e5eb68763-tmp\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.115387 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7e2afe34-c831-4376-a249-119e5eb68763-snapshots\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.115442 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.115507 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert podName:ca0bbe23-8ace-42cd-8357-552e69aa9648 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:39.615490919 +0000 UTC m=+237.825119361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-dx56d" (UID: "ca0bbe23-8ace-42cd-8357-552e69aa9648") : secret "networking-console-plugin-cert" not found Aug 01 00:37:39.115576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.115523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e2afe34-c831-4376-a249-119e5eb68763-service-ca-bundle\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.116266 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.115688 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e2afe34-c831-4376-a249-119e5eb68763-trusted-ca-bundle\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.117142 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.117118 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7e2afe34-c831-4376-a249-119e5eb68763-serving-cert\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.118013 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.117967 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.125891 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.125860 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxjrz\" (UniqueName: \"kubernetes.io/projected/7e2afe34-c831-4376-a249-119e5eb68763-kube-api-access-pxjrz\") pod \"insights-operator-77696685b-zmdst\" (UID: \"7e2afe34-c831-4376-a249-119e5eb68763\") " pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.126793 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.126762 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jts\" (UniqueName: \"kubernetes.io/projected/f7b5f441-b04a-4b1c-9b75-9bf091c54a21-kube-api-access-d6jts\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6gx8p\" (UID: \"f7b5f441-b04a-4b1c-9b75-9bf091c54a21\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.126854 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.126811 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bl22\" (UniqueName: \"kubernetes.io/projected/d79164a3-2f9d-49b2-a10a-baca631e83a2-kube-api-access-6bl22\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:39.127468 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.127454 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" Aug 01 00:37:39.135852 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.135823 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-zmdst" Aug 01 00:37:39.148541 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.148512 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5"] Aug 01 00:37:39.152231 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:37:39.152187 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143164f9_1292_4b96_9f5f_1b3958ae6f8d.slice/crio-c9ccb3ffe4d92bdc4d503083a9227c38c9ecb002ea63dfb341b9b00ffd48112c WatchSource:0}: Error finding container c9ccb3ffe4d92bdc4d503083a9227c38c9ecb002ea63dfb341b9b00ffd48112c: Status 404 returned error can't find the container with id c9ccb3ffe4d92bdc4d503083a9227c38c9ecb002ea63dfb341b9b00ffd48112c Aug 01 00:37:39.169715 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.169621 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg"] Aug 01 00:37:39.175629 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:37:39.175594 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976974b1_114c_4c8e_96ce_86a0a2fb7552.slice/crio-746100320a09bcacf7d6e5273f7bb4b2f0d279343dce6c3702b7e0f96c5cc663 WatchSource:0}: Error finding container 746100320a09bcacf7d6e5273f7bb4b2f0d279343dce6c3702b7e0f96c5cc663: Status 404 returned error can't find the container with id 746100320a09bcacf7d6e5273f7bb4b2f0d279343dce6c3702b7e0f96c5cc663 Aug 01 00:37:39.215471 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.215252 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" Aug 01 00:37:39.278526 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.278439 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz"] Aug 01 00:37:39.283236 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:37:39.283175 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac70d06_1990_4dbf_83ee_f221288845aa.slice/crio-feead505abdc1e4c0340c9489193055345f323e9588fbb01794338915a31c04c WatchSource:0}: Error finding container feead505abdc1e4c0340c9489193055345f323e9588fbb01794338915a31c04c: Status 404 returned error can't find the container with id feead505abdc1e4c0340c9489193055345f323e9588fbb01794338915a31c04c Aug 01 00:37:39.304053 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.304022 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-zmdst"] Aug 01 00:37:39.306798 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:37:39.306767 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2afe34_c831_4376_a249_119e5eb68763.slice/crio-2dd32fca7ad0920d271a0d0c3c30394ad003ab6294bbd5c59f112e78ab90eb9d WatchSource:0}: Error finding container 2dd32fca7ad0920d271a0d0c3c30394ad003ab6294bbd5c59f112e78ab90eb9d: Status 404 returned error can't find the container with id 2dd32fca7ad0920d271a0d0c3c30394ad003ab6294bbd5c59f112e78ab90eb9d Aug 01 00:37:39.360226 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.360182 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p"] Aug 01 00:37:39.364477 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:37:39.364453 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b5f441_b04a_4b1c_9b75_9bf091c54a21.slice/crio-26947ad922c546dd3e9ff999166659908dec4c4a7510a8921dc3d0c72d3541e1 WatchSource:0}: Error finding container 26947ad922c546dd3e9ff999166659908dec4c4a7510a8921dc3d0c72d3541e1: Status 404 returned error can't find the container with id 26947ad922c546dd3e9ff999166659908dec4c4a7510a8921dc3d0c72d3541e1 Aug 01 00:37:39.517780 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.517743 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:39.517947 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.517852 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.517947 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.517877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:39.517947 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.517907 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:39.518122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.517980 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 01 00:37:39.518122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.517986 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls podName:b1fac243-00b1-4b7d-a1c9-5ba809a41789 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:40.517965679 +0000 UTC m=+238.727594130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-f7lwh" (UID: "b1fac243-00b1-4b7d-a1c9-5ba809a41789") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:39.518122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.518031 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:40.518019686 +0000 UTC m=+238.727648124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : configmap references non-existent config key: service-ca.crt Aug 01 00:37:39.518122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.518043 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:40.518036868 +0000 UTC m=+238.727665303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : secret "router-metrics-certs-default" not found Aug 01 00:37:39.619205 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.619110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:39.619369 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.619241 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:39.619369 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.619289 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 01 00:37:39.619369 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.619366 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert podName:ca0bbe23-8ace-42cd-8357-552e69aa9648 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:40.619344991 +0000 UTC m=+238.828973444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-dx56d" (UID: "ca0bbe23-8ace-42cd-8357-552e69aa9648") : secret "networking-console-plugin-cert" not found Aug 01 00:37:39.619543 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.619381 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:37:39.619543 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:39.619430 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls podName:d79164a3-2f9d-49b2-a10a-baca631e83a2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:40.619415367 +0000 UTC m=+238.829043808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-m6qzj" (UID: "d79164a3-2f9d-49b2-a10a-baca631e83a2") : secret "samples-operator-tls" not found Aug 01 00:37:39.891611 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.891518 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-zmdst" event={"ID":"7e2afe34-c831-4376-a249-119e5eb68763","Type":"ContainerStarted","Data":"2dd32fca7ad0920d271a0d0c3c30394ad003ab6294bbd5c59f112e78ab90eb9d"} Aug 01 00:37:39.892689 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.892657 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg" event={"ID":"976974b1-114c-4c8e-96ce-86a0a2fb7552","Type":"ContainerStarted","Data":"746100320a09bcacf7d6e5273f7bb4b2f0d279343dce6c3702b7e0f96c5cc663"} Aug 01 00:37:39.894460 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.894340 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5" event={"ID":"143164f9-1292-4b96-9f5f-1b3958ae6f8d","Type":"ContainerStarted","Data":"2cf80b80fd139806814a411a53640f3edd28eaaaf4fcfbeb5af037687854dc6d"} Aug 01 00:37:39.894460 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.894391 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5" event={"ID":"143164f9-1292-4b96-9f5f-1b3958ae6f8d","Type":"ContainerStarted","Data":"c9ccb3ffe4d92bdc4d503083a9227c38c9ecb002ea63dfb341b9b00ffd48112c"} Aug 01 00:37:39.895800 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.895767 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" event={"ID":"f7b5f441-b04a-4b1c-9b75-9bf091c54a21","Type":"ContainerStarted","Data":"26947ad922c546dd3e9ff999166659908dec4c4a7510a8921dc3d0c72d3541e1"} Aug 01 00:37:39.899992 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.899954 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" event={"ID":"cac70d06-1990-4dbf-83ee-f221288845aa","Type":"ContainerStarted","Data":"feead505abdc1e4c0340c9489193055345f323e9588fbb01794338915a31c04c"} Aug 01 00:37:39.911615 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:39.911556 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-jfpj5" podStartSLOduration=1.911537558 podStartE2EDuration="1.911537558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:39.910451059 +0000 UTC m=+238.120079517" watchObservedRunningTime="2026-08-01 00:37:39.911537558 +0000 UTC m=+238.121166016" Aug 01 00:37:40.535423 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:40.535377 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:40.535598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:40.535448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:40.535598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:40.535490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:40.535705 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.535644 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:40.535759 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.535706 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls podName:b1fac243-00b1-4b7d-a1c9-5ba809a41789 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:42.535688701 +0000 UTC m=+240.745317142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-f7lwh" (UID: "b1fac243-00b1-4b7d-a1c9-5ba809a41789") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:40.536122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.536102 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 01 00:37:40.536255 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.536153 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:42.536139733 +0000 UTC m=+240.745768171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : secret "router-metrics-certs-default" not found Aug 01 00:37:40.536255 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.536231 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:42.536202707 +0000 UTC m=+240.745831146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : configmap references non-existent config key: service-ca.crt Aug 01 00:37:40.636279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:40.636243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:40.636481 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:40.636390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:40.636549 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.636512 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 01 00:37:40.636601 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.636575 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert podName:ca0bbe23-8ace-42cd-8357-552e69aa9648 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:42.636556822 +0000 UTC m=+240.846185260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-dx56d" (UID: "ca0bbe23-8ace-42cd-8357-552e69aa9648") : secret "networking-console-plugin-cert" not found Aug 01 00:37:40.636662 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.636639 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:37:40.636709 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:40.636701 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls podName:d79164a3-2f9d-49b2-a10a-baca631e83a2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:42.636684347 +0000 UTC m=+240.846312787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-m6qzj" (UID: "d79164a3-2f9d-49b2-a10a-baca631e83a2") : secret "samples-operator-tls" not found Aug 01 00:37:40.906300 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:40.906161 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg" event={"ID":"976974b1-114c-4c8e-96ce-86a0a2fb7552","Type":"ContainerStarted","Data":"c9ed3a3abe4216b86dd01cee293a1896454188f25349988f7ede58d6c605b3c1"} Aug 01 00:37:40.923040 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:40.922869 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-jrrzg" podStartSLOduration=2.008471705 podStartE2EDuration="2.92284365s" totalImagesPullingTime="914.371945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:40.921096295 +0000 UTC m=+239.130724754" watchObservedRunningTime="2026-08-01 00:37:40.92284365 +0000 UTC m=+239.132472109" Aug 01 00:37:42.553234 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.553189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:42.553533 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.553320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:42.553533 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.553323 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:42.553533 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.553346 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:42.553533 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.553381 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls podName:b1fac243-00b1-4b7d-a1c9-5ba809a41789 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:46.553366622 +0000 UTC m=+244.762995057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-f7lwh" (UID: "b1fac243-00b1-4b7d-a1c9-5ba809a41789") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:42.553533 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.553459 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 01 00:37:42.553533 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.553474 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:46.553458345 +0000 UTC m=+244.763086780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : configmap references non-existent config key: service-ca.crt Aug 01 00:37:42.553533 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.553497 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:46.553486696 +0000 UTC m=+244.763115150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : secret "router-metrics-certs-default" not found Aug 01 00:37:42.653865 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.653817 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:42.654059 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.653986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:42.654059 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.653999 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:37:42.654173 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.654072 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls podName:d79164a3-2f9d-49b2-a10a-baca631e83a2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:46.654051142 +0000 UTC m=+244.863679591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-m6qzj" (UID: "d79164a3-2f9d-49b2-a10a-baca631e83a2") : secret "samples-operator-tls" not found Aug 01 00:37:42.654173 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.654111 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 01 00:37:42.654173 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:42.654170 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert podName:ca0bbe23-8ace-42cd-8357-552e69aa9648 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:46.654153043 +0000 UTC m=+244.863781496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-dx56d" (UID: "ca0bbe23-8ace-42cd-8357-552e69aa9648") : secret "networking-console-plugin-cert" not found Aug 01 00:37:42.913591 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.913505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" event={"ID":"cac70d06-1990-4dbf-83ee-f221288845aa","Type":"ContainerStarted","Data":"81a5cdaa46ef9356a822b6f8a61c11d666321df7ee54cfa288f7ea502cc28ee0"} Aug 01 00:37:42.915369 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.915188 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-zmdst" event={"ID":"7e2afe34-c831-4376-a249-119e5eb68763","Type":"ContainerStarted","Data":"b07322ce8723924c267dc07d047ebd542d70cea13a9eb7fb5a28f75b4bc38879"} Aug 01 00:37:42.916725 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.916698 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" event={"ID":"f7b5f441-b04a-4b1c-9b75-9bf091c54a21","Type":"ContainerStarted","Data":"2fc03602624202dbe97d30839319ad6d3e129ba176e8b392382d24ab0913ac36"} Aug 01 00:37:42.928965 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.928912 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" podStartSLOduration=1.6854051829999999 podStartE2EDuration="4.928894056s" totalImagesPullingTime="3.243488873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:42.927925263 +0000 UTC m=+241.137553719" watchObservedRunningTime="2026-08-01 00:37:42.928894056 +0000 UTC m=+241.138522516" Aug 01 00:37:42.961289 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.961238 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-zmdst" podStartSLOduration=1.7468634490000001 podStartE2EDuration="4.961198632s" totalImagesPullingTime="3.214335183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:42.960187027 +0000 UTC m=+241.169815485" watchObservedRunningTime="2026-08-01 00:37:42.961198632 +0000 UTC m=+241.170827088" Aug 01 00:37:42.961562 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:42.961535 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" podStartSLOduration=1.80733035 podStartE2EDuration="4.961526768s" totalImagesPullingTime="3.154196418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:42.943843666 +0000 UTC m=+241.153472125" watchObservedRunningTime="2026-08-01 00:37:42.961526768 +0000 UTC m=+241.171155225" Aug 01 00:37:44.260248 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.260197 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6cpp8"] Aug 01 00:37:44.264502 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.264477 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.266560 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.266538 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 01 00:37:44.266680 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.266558 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 01 00:37:44.266680 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.266562 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jls8z\"" Aug 01 00:37:44.273937 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.273898 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6cpp8"] Aug 01 00:37:44.369201 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.369164 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.369201 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.369203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.369430 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.369406 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-data-volume\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.369470 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.369448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-crio-socket\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.369503 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.369474 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rtz\" (UniqueName: \"kubernetes.io/projected/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-api-access-j2rtz\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.470714 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.470676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-data-volume\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.470876 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.470722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-crio-socket\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.470876 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.470740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rtz\" (UniqueName: \"kubernetes.io/projected/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-api-access-j2rtz\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.470876 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.470758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.470876 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.470775 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.470876 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.470849 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-crio-socket\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.471034 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:44.470872 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:37:44.471034 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:44.470883 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:37:44.471034 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:44.470942 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:37:44.970922584 +0000 UTC m=+243.180551020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : configmap "kube-rbac-proxy" not found Aug 01 00:37:44.471034 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:44.470956 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:37:44.970949859 +0000 UTC m=+243.180578294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : secret "insights-runtime-extractor-tls" not found Aug 01 00:37:44.471163 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.471082 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-data-volume\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.478444 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.478420 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rtz\" (UniqueName: \"kubernetes.io/projected/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-api-access-j2rtz\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.974685 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.974646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.974685 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:44.974685 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:44.974898 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:44.974775 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:37:44.974898 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:44.974838 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:37:45.974823053 +0000 UTC m=+244.184451488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : configmap "kube-rbac-proxy" not found Aug 01 00:37:44.974898 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:44.974779 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:37:44.974995 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:44.974922 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:37:45.974910921 +0000 UTC m=+244.184539360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : secret "insights-runtime-extractor-tls" not found Aug 01 00:37:45.667886 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.667852 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-k7w42"] Aug 01 00:37:45.671065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.671046 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.673340 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.673312 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 01 00:37:45.673451 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.673321 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 01 00:37:45.673451 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.673382 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 01 00:37:45.673451 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.673387 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 01 00:37:45.674029 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.674016 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6pls7\"" Aug 01 00:37:45.680393 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.680371 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-k7w42"] Aug 01 00:37:45.781134 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.781094 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e326f486-3807-4893-8451-e8a5b4bf469c-signing-cabundle\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.781336 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.781165 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e326f486-3807-4893-8451-e8a5b4bf469c-signing-key\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.781336 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.781269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5wc\" (UniqueName: \"kubernetes.io/projected/e326f486-3807-4893-8451-e8a5b4bf469c-kube-api-access-tl5wc\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.881842 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.881804 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e326f486-3807-4893-8451-e8a5b4bf469c-signing-cabundle\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.882037 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.881866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e326f486-3807-4893-8451-e8a5b4bf469c-signing-key\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.882098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.882049 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5wc\" (UniqueName: \"kubernetes.io/projected/e326f486-3807-4893-8451-e8a5b4bf469c-kube-api-access-tl5wc\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.882567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.882545 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e326f486-3807-4893-8451-e8a5b4bf469c-signing-cabundle\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.884506 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.884484 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e326f486-3807-4893-8451-e8a5b4bf469c-signing-key\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.889523 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.889503 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5wc\" (UniqueName: \"kubernetes.io/projected/e326f486-3807-4893-8451-e8a5b4bf469c-kube-api-access-tl5wc\") pod \"service-ca-98c88fbb5-k7w42\" (UID: \"e326f486-3807-4893-8451-e8a5b4bf469c\") " pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.980359 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.980312 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" Aug 01 00:37:45.983075 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.983044 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:45.983225 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:45.983090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:45.983225 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:45.983185 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:37:45.983339 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:45.983205 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:37:45.983339 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:45.983298 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:37:47.983278496 +0000 UTC m=+246.192906950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : secret "insights-runtime-extractor-tls" not found Aug 01 00:37:45.983339 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:45.983329 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:37:47.983318962 +0000 UTC m=+246.192947404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : configmap "kube-rbac-proxy" not found Aug 01 00:37:46.109227 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.109184 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-k7w42"] Aug 01 00:37:46.113205 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:37:46.112493 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode326f486_3807_4893_8451_e8a5b4bf469c.slice/crio-7c7dd342410d7347362eb6cc53c12af5bf3320f223f27ff427cd7a24006fe8f7 WatchSource:0}: Error finding container 7c7dd342410d7347362eb6cc53c12af5bf3320f223f27ff427cd7a24006fe8f7: Status 404 returned error can't find the container with id 7c7dd342410d7347362eb6cc53c12af5bf3320f223f27ff427cd7a24006fe8f7 Aug 01 00:37:46.589055 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.589021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:46.589251 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.589062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:46.589251 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.589176 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 01 00:37:46.589251 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.589187 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:54.589171449 +0000 UTC m=+252.798799899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : configmap references non-existent config key: service-ca.crt Aug 01 00:37:46.589368 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.589308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:46.589368 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.589350 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs podName:5027f8b0-fdc9-48e1-bdd1-898dafd7fae2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:54.589332726 +0000 UTC m=+252.798961176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs") pod "router-default-64956b7558-rqmzr" (UID: "5027f8b0-fdc9-48e1-bdd1-898dafd7fae2") : secret "router-metrics-certs-default" not found Aug 01 00:37:46.589430 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.589409 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:46.589461 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.589456 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls podName:b1fac243-00b1-4b7d-a1c9-5ba809a41789 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:54.589440463 +0000 UTC m=+252.799068900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-f7lwh" (UID: "b1fac243-00b1-4b7d-a1c9-5ba809a41789") : secret "cluster-monitoring-operator-tls" not found Aug 01 00:37:46.690569 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.690531 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:46.691044 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.690600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:46.691044 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.690687 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 01 00:37:46.691044 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.690707 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 01 00:37:46.691044 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.690752 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert podName:ca0bbe23-8ace-42cd-8357-552e69aa9648 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:54.690738216 +0000 UTC m=+252.900366650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-dx56d" (UID: "ca0bbe23-8ace-42cd-8357-552e69aa9648") : secret "networking-console-plugin-cert" not found Aug 01 00:37:46.691044 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:46.690767 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls podName:d79164a3-2f9d-49b2-a10a-baca631e83a2 nodeName:}" failed. No retries permitted until 2026-08-01 00:37:54.690761369 +0000 UTC m=+252.900389804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-m6qzj" (UID: "d79164a3-2f9d-49b2-a10a-baca631e83a2") : secret "samples-operator-tls" not found Aug 01 00:37:46.931007 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.930974 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" event={"ID":"e326f486-3807-4893-8451-e8a5b4bf469c","Type":"ContainerStarted","Data":"d2bdb3baa5184fdedbfb7e78491d06db08f9d0b642b32ebb3c14e7e21ec43c4b"} Aug 01 00:37:46.931182 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.931029 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" event={"ID":"e326f486-3807-4893-8451-e8a5b4bf469c","Type":"ContainerStarted","Data":"7c7dd342410d7347362eb6cc53c12af5bf3320f223f27ff427cd7a24006fe8f7"} Aug 01 00:37:46.949614 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:46.949514 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-k7w42" podStartSLOduration=1.94949386 podStartE2EDuration="1.94949386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:46.949037167 +0000 UTC m=+245.158665626" watchObservedRunningTime="2026-08-01 00:37:46.94949386 +0000 UTC m=+245.159122321" Aug 01 00:37:47.237883 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:47.237801 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4mkf_10cbd086-8ace-48e5-bda2-915b5ba02111/dns-node-resolver/0.log" Aug 01 00:37:48.003738 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:48.003684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:48.004122 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:48.003752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:48.004122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:48.003755 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:37:48.004122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:48.003809 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:37:48.004122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:48.003913 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:37:52.003886061 +0000 UTC m=+250.213514515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : configmap "kube-rbac-proxy" not found Aug 01 00:37:48.004122 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:48.003946 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:37:52.00392832 +0000 UTC m=+250.213556780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : secret "insights-runtime-extractor-tls" not found Aug 01 00:37:48.240252 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:48.240224 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vm2k4_06e31806-756d-4782-9193-ac8ce656d17b/node-ca/0.log" Aug 01 00:37:49.040688 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:49.040608 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kvg6c_8cdf8609-6802-4dc9-b5b6-ca974534987b/serve-healthcheck-canary/0.log" Aug 01 00:37:49.640861 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:49.640825 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-6gx8p_f7b5f441-b04a-4b1c-9b75-9bf091c54a21/kube-storage-version-migrator-operator/0.log" Aug 01 00:37:52.038430 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:52.038382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:52.038430 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:52.038426 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:37:52.038864 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:52.038525 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:37:52.038864 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:52.038598 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:38:00.038579211 +0000 UTC m=+258.248207647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : configmap "kube-rbac-proxy" not found Aug 01 00:37:52.038864 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:52.038648 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:37:52.038864 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:52.038702 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls podName:d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b nodeName:}" failed. No retries permitted until 2026-08-01 00:38:00.03868832 +0000 UTC m=+258.248316754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cpp8" (UID: "d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b") : secret "insights-runtime-extractor-tls" not found Aug 01 00:37:53.047617 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:53.047572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:37:53.048016 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:53.047716 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 01 00:37:53.048016 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:53.047780 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs podName:dcf38574-98e0-464c-8f21-bdb577e4b0ec nodeName:}" failed. No retries permitted until 2026-08-01 00:39:55.047763831 +0000 UTC m=+373.257392269 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs") pod "network-metrics-daemon-249m9" (UID: "dcf38574-98e0-464c-8f21-bdb577e4b0ec") : secret "metrics-daemon-secret" not found Aug 01 00:37:54.663364 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.663317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:54.663364 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.663373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:54.663923 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.663417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:54.664132 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.664106 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-service-ca-bundle\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:54.665885 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.665861 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5027f8b0-fdc9-48e1-bdd1-898dafd7fae2-metrics-certs\") pod \"router-default-64956b7558-rqmzr\" (UID: \"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2\") " pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:54.665944 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.665930 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1fac243-00b1-4b7d-a1c9-5ba809a41789-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-f7lwh\" (UID: \"b1fac243-00b1-4b7d-a1c9-5ba809a41789\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:54.764299 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.764263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:37:54.764467 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.764325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:54.764467 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:54.764440 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 01 00:37:54.764549 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:37:54.764519 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert podName:ca0bbe23-8ace-42cd-8357-552e69aa9648 nodeName:}" failed. No retries permitted until 2026-08-01 00:38:10.764499283 +0000 UTC m=+268.974127722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-dx56d" (UID: "ca0bbe23-8ace-42cd-8357-552e69aa9648") : secret "networking-console-plugin-cert" not found Aug 01 00:37:54.766950 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.766925 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d79164a3-2f9d-49b2-a10a-baca631e83a2-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-m6qzj\" (UID: \"d79164a3-2f9d-49b2-a10a-baca631e83a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:54.933905 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.933877 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j5cqs\"" Aug 01 00:37:54.940091 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.940065 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rr59w\"" Aug 01 00:37:54.942626 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.942601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" Aug 01 00:37:54.947783 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:54.947761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:55.043225 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.043185 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7n46z\"" Aug 01 00:37:55.051770 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.051747 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" Aug 01 00:37:55.098951 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.098927 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh"] Aug 01 00:37:55.102907 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:37:55.102880 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fac243_00b1_4b7d_a1c9_5ba809a41789.slice/crio-29ab14ba0d21215928296b2a27d2c751e0e008b1615a8f9f8fa6107c7599000b WatchSource:0}: Error finding container 29ab14ba0d21215928296b2a27d2c751e0e008b1615a8f9f8fa6107c7599000b: Status 404 returned error can't find the container with id 29ab14ba0d21215928296b2a27d2c751e0e008b1615a8f9f8fa6107c7599000b Aug 01 00:37:55.117545 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.117093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64956b7558-rqmzr"] Aug 01 00:37:55.122704 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:37:55.122662 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5027f8b0_fdc9_48e1_bdd1_898dafd7fae2.slice/crio-ce7180d4c7dd406f371ded471fd81d62341995b6a868bf4679cba090114224e1 WatchSource:0}: Error finding container ce7180d4c7dd406f371ded471fd81d62341995b6a868bf4679cba090114224e1: Status 404 returned error can't find the container with id ce7180d4c7dd406f371ded471fd81d62341995b6a868bf4679cba090114224e1 Aug 01 00:37:55.206485 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.206457 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj"] Aug 01 00:37:55.964198 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.964161 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" event={"ID":"d79164a3-2f9d-49b2-a10a-baca631e83a2","Type":"ContainerStarted","Data":"596aef8b78bfcd94e7cef4bd641ba98a46ab3a1448c8b3760d2960642e2d683e"} Aug 01 00:37:55.966042 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.966011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" event={"ID":"b1fac243-00b1-4b7d-a1c9-5ba809a41789","Type":"ContainerStarted","Data":"29ab14ba0d21215928296b2a27d2c751e0e008b1615a8f9f8fa6107c7599000b"} Aug 01 00:37:55.968389 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.968358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64956b7558-rqmzr" event={"ID":"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2","Type":"ContainerStarted","Data":"3c9c867ab04728b85d0fad0846784c0242155ea053d331c566a7cfc76af9b889"} Aug 01 00:37:55.968513 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.968410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64956b7558-rqmzr" event={"ID":"5027f8b0-fdc9-48e1-bdd1-898dafd7fae2","Type":"ContainerStarted","Data":"ce7180d4c7dd406f371ded471fd81d62341995b6a868bf4679cba090114224e1"} Aug 01 00:37:55.987180 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:55.987130 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64956b7558-rqmzr" podStartSLOduration=17.987109565 podStartE2EDuration="17.987109565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:55.986230156 +0000 UTC m=+254.195858605" watchObservedRunningTime="2026-08-01 00:37:55.987109565 +0000 UTC m=+254.196738023" Aug 01 00:37:56.948624 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:56.948590 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:56.951537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:56.951515 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:56.977843 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:56.977795 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:56.979162 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:56.979128 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64956b7558-rqmzr" Aug 01 00:37:57.982884 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:57.982849 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" event={"ID":"d79164a3-2f9d-49b2-a10a-baca631e83a2","Type":"ContainerStarted","Data":"12d19c7b037d7e71722c9d0022c042a7222e61a593a2876ee2c2eb1977651d3d"} Aug 01 00:37:57.983358 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:57.982909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" event={"ID":"d79164a3-2f9d-49b2-a10a-baca631e83a2","Type":"ContainerStarted","Data":"006e03c4478db7127d08af23fe74689bb8235fc14673e763207d247d02590a39"} Aug 01 00:37:57.984332 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:57.984306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" event={"ID":"b1fac243-00b1-4b7d-a1c9-5ba809a41789","Type":"ContainerStarted","Data":"4032653dbaeee801c08f51d1aca6f1ee883d06e4a4e759eb765ccfe42b0da83c"} Aug 01 00:37:57.999735 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:57.999691 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-m6qzj" podStartSLOduration=17.920058122 podStartE2EDuration="19.99967813s" totalImagesPullingTime="2.079620008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:57.997994198 +0000 UTC m=+256.207622655" watchObservedRunningTime="2026-08-01 00:37:57.99967813 +0000 UTC m=+256.209306587" Aug 01 00:37:58.015999 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:37:58.015953 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-f7lwh" podStartSLOduration=17.782043067 podStartE2EDuration="20.015938732s" totalImagesPullingTime="2.233895665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:58.015257974 +0000 UTC m=+256.224886434" watchObservedRunningTime="2026-08-01 00:37:58.015938732 +0000 UTC m=+256.225567189" Aug 01 00:38:00.114252 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:00.114193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:38:00.114252 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:00.114254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:38:00.114795 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:00.114725 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:38:00.116737 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:00.116715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cpp8\" (UID: \"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b\") " pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:38:00.174203 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:00.174161 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6cpp8" Aug 01 00:38:00.310794 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:00.310762 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6cpp8"] Aug 01 00:38:00.314201 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:38:00.314174 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ebc20c_a592_4fb3_b8fe_b4f2924d9d0b.slice/crio-ad578ef48cd2a3e3ed351190125e0784a5bc40ec7e91b5fa89848918bc01c734 WatchSource:0}: Error finding container ad578ef48cd2a3e3ed351190125e0784a5bc40ec7e91b5fa89848918bc01c734: Status 404 returned error can't find the container with id ad578ef48cd2a3e3ed351190125e0784a5bc40ec7e91b5fa89848918bc01c734 Aug 01 00:38:00.994560 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:00.994522 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cpp8" event={"ID":"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b","Type":"ContainerStarted","Data":"e8fcca4ce88a36b86b913ad83230db7cb85e718dbb7bdee5eebe6943bdfe8126"} Aug 01 00:38:00.994747 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:00.994593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cpp8" event={"ID":"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b","Type":"ContainerStarted","Data":"ad578ef48cd2a3e3ed351190125e0784a5bc40ec7e91b5fa89848918bc01c734"} Aug 01 00:38:02.002129 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:02.002092 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cpp8" event={"ID":"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b","Type":"ContainerStarted","Data":"20bb1da7c13e366c71afdab41685ac111b950ff6bb8cd3f600a75e63e59b5049"} Aug 01 00:38:03.006895 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:03.006851 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cpp8" event={"ID":"d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b","Type":"ContainerStarted","Data":"5060bd33332925ebb953701f72e2c6f8cb0fd8fe812d66bd113b5283df326b97"} Aug 01 00:38:03.036909 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:03.036853 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6cpp8" podStartSLOduration=16.72118604 podStartE2EDuration="19.036837513s" totalImagesPullingTime="2.315651473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:37:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:03.035940776 +0000 UTC m=+261.245569233" watchObservedRunningTime="2026-08-01 00:38:03.036837513 +0000 UTC m=+261.246465969" Aug 01 00:38:10.804738 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:10.804689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:38:10.807492 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:10.807456 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca0bbe23-8ace-42cd-8357-552e69aa9648-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-dx56d\" (UID: \"ca0bbe23-8ace-42cd-8357-552e69aa9648\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:38:10.948263 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:10.948201 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-44cbq\"" Aug 01 00:38:10.956201 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:10.956174 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" Aug 01 00:38:11.090851 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:11.090771 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-dx56d"] Aug 01 00:38:11.094076 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:38:11.094047 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0bbe23_8ace_42cd_8357_552e69aa9648.slice/crio-cab608ba6b1ce308c8e8161c90896205ab09b9bcb0d6e09bbd2a44ef50f05afd WatchSource:0}: Error finding container cab608ba6b1ce308c8e8161c90896205ab09b9bcb0d6e09bbd2a44ef50f05afd: Status 404 returned error can't find the container with id cab608ba6b1ce308c8e8161c90896205ab09b9bcb0d6e09bbd2a44ef50f05afd Aug 01 00:38:12.038262 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.038224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" event={"ID":"ca0bbe23-8ace-42cd-8357-552e69aa9648","Type":"ContainerStarted","Data":"cab608ba6b1ce308c8e8161c90896205ab09b9bcb0d6e09bbd2a44ef50f05afd"} Aug 01 00:38:12.231595 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.230742 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v745r"] Aug 01 00:38:12.238116 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.238079 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.242182 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.241927 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 01 00:38:12.242182 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.242038 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nvkkd\"" Aug 01 00:38:12.242444 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.242421 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 01 00:38:12.243050 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.242636 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 01 00:38:12.243050 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.242855 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 01 00:38:12.317870 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.317771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-sys\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.317870 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.317841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-accelerators-collector-config\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.318103 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.317902 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-tls\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.318103 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.317959 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv86\" (UniqueName: \"kubernetes.io/projected/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-kube-api-access-tjv86\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.318103 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.317983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-textfile\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.318103 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.318031 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-wtmp\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.318103 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.318046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-metrics-client-ca\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.318103 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.318069 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-root\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.318422 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.318127 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.421037 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-sys\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.421198 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-accelerators-collector-config\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.421198 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-tls\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.421198 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv86\" (UniqueName: \"kubernetes.io/projected/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-kube-api-access-tjv86\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.421198 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421139 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-sys\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.421198 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421163 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-textfile\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-wtmp\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:38:12.421541 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-metrics-client-ca\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:38:12.421604 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-tls podName:d25f9979-ce83-4d01-9b0b-aabb2ddc7c68 nodeName:}" failed. No retries permitted until 2026-08-01 00:38:12.921584843 +0000 UTC m=+271.131213278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-tls") pod "node-exporter-v745r" (UID: "d25f9979-ce83-4d01-9b0b-aabb2ddc7c68") : secret "node-exporter-tls" not found Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421638 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-root\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421719 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421757 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-accelerators-collector-config\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421805 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-textfile\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421892 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-wtmp\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422020 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.421935 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-root\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.422870 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.422050 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-metrics-client-ca\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.424743 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.424714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.433745 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.433718 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv86\" (UniqueName: \"kubernetes.io/projected/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-kube-api-access-tjv86\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.926774 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.926733 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-tls\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:12.929291 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:12.929253 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25f9979-ce83-4d01-9b0b-aabb2ddc7c68-node-exporter-tls\") pod \"node-exporter-v745r\" (UID: \"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68\") " pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:13.042786 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.042751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" event={"ID":"ca0bbe23-8ace-42cd-8357-552e69aa9648","Type":"ContainerStarted","Data":"218ee1423ff5704ab2f072555416e971014d69beb8c67b874c3ada76840d2994"} Aug 01 00:38:13.056420 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.056353 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-dx56d" podStartSLOduration=33.756388136 podStartE2EDuration="35.056327318s" totalImagesPullingTime="1.299939182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:13.055582566 +0000 UTC m=+271.265211023" watchObservedRunningTime="2026-08-01 00:38:13.056327318 +0000 UTC m=+271.265955775" Aug 01 00:38:13.153359 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.153310 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v745r" Aug 01 00:38:13.164413 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:38:13.164385 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25f9979_ce83_4d01_9b0b_aabb2ddc7c68.slice/crio-a2e7b01e1cc9a64e7be36f84dc55066adfd9335d49bc1cc38d017df92fa1fc7c WatchSource:0}: Error finding container a2e7b01e1cc9a64e7be36f84dc55066adfd9335d49bc1cc38d017df92fa1fc7c: Status 404 returned error can't find the container with id a2e7b01e1cc9a64e7be36f84dc55066adfd9335d49bc1cc38d017df92fa1fc7c Aug 01 00:38:13.290497 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.290458 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:38:13.295369 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.295347 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.297573 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.297521 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 01 00:38:13.297716 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.297581 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 01 00:38:13.297716 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.297627 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 01 00:38:13.297716 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.297535 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zrnxp\"" Aug 01 00:38:13.297872 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.297732 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 01 00:38:13.297872 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.297774 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 01 00:38:13.297872 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.297804 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 01 00:38:13.297872 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.297841 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 01 00:38:13.298147 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.298130 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 01 00:38:13.298247 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.298138 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 01 00:38:13.310126 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.310100 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:38:13.432329 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432294 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432329 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432564 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432564 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432564 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-config-volume\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432564 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432564 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-config-out\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432564 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432477 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432564 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432774 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432774 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432621 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94pn\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-kube-api-access-d94pn\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432774 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432683 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.432774 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.432717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-web-config\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.533780 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.533732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.533974 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.533787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.533974 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.533839 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.533974 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.533866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.533974 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.533893 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-config-volume\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.533974 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.533925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.533974 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.533959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-config-out\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.534308 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.533982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.534308 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.534038 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.534308 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.534136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.534308 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.534183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94pn\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-kube-api-access-d94pn\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.534308 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.534243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.534308 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.534274 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-web-config\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.534308 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.534292 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.535232 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.535180 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.537939 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.536869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.537939 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.537472 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.537939 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.537886 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.538237 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.538196 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.538303 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.537633 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-config-out\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.538950 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.538924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-web-config\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.539142 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.539097 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-config-volume\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.539269 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.539161 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.539542 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.539520 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.540319 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.540277 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.545850 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.545790 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94pn\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-kube-api-access-d94pn\") pod \"alertmanager-main-0\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.605226 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.605173 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:38:13.753493 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:13.753455 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:38:13.759498 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:38:13.759464 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd742d153_2e24_4cb2_9422_275facf5e706.slice/crio-71799f90435e3e6a2f1334bc46131b1afe849015f5096181b8cec3407a770ece WatchSource:0}: Error finding container 71799f90435e3e6a2f1334bc46131b1afe849015f5096181b8cec3407a770ece: Status 404 returned error can't find the container with id 71799f90435e3e6a2f1334bc46131b1afe849015f5096181b8cec3407a770ece Aug 01 00:38:14.049301 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:14.049263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerStarted","Data":"71799f90435e3e6a2f1334bc46131b1afe849015f5096181b8cec3407a770ece"} Aug 01 00:38:14.050682 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:14.050650 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v745r" event={"ID":"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68","Type":"ContainerStarted","Data":"a2e7b01e1cc9a64e7be36f84dc55066adfd9335d49bc1cc38d017df92fa1fc7c"} Aug 01 00:38:15.056664 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.056628 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerStarted","Data":"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b"} Aug 01 00:38:15.058466 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.058371 2568 generic.go:356] "Generic (PLEG): container finished" podID="d25f9979-ce83-4d01-9b0b-aabb2ddc7c68" containerID="ec74d2b84620656ae7f0a37c533c5540e6145b6e1ef46d91deb41e52691b0bb6" exitCode=0 Aug 01 00:38:15.058531 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.058453 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v745r" event={"ID":"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68","Type":"ContainerDied","Data":"ec74d2b84620656ae7f0a37c533c5540e6145b6e1ef46d91deb41e52691b0bb6"} Aug 01 00:38:15.225401 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.225360 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84864f89c5-7zbl8"] Aug 01 00:38:15.233918 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.233885 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.236312 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.236275 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 01 00:38:15.236475 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.236456 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ct8kq\"" Aug 01 00:38:15.236537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.236485 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 01 00:38:15.236537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.236496 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 01 00:38:15.236537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.236504 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-40ioks6j8l4c2\"" Aug 01 00:38:15.236693 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.236456 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 01 00:38:15.236800 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.236785 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 01 00:38:15.244451 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.244425 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84864f89c5-7zbl8"] Aug 01 00:38:15.352728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.352690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.353087 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.352779 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.353087 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.352820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.353087 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.352897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-tls\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.353087 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.352935 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-grpc-tls\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.353087 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.352951 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6n2f\" (UniqueName: \"kubernetes.io/projected/cda0c369-3d5b-47d2-a9e5-1962a873c20d-kube-api-access-l6n2f\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.353087 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.352969 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda0c369-3d5b-47d2-a9e5-1962a873c20d-metrics-client-ca\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.353087 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.352997 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.453588 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.453490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.453588 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.453543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.453806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.453594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-tls\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.453806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.453658 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-grpc-tls\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.453806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.453690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6n2f\" (UniqueName: \"kubernetes.io/projected/cda0c369-3d5b-47d2-a9e5-1962a873c20d-kube-api-access-l6n2f\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.453806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.453709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda0c369-3d5b-47d2-a9e5-1962a873c20d-metrics-client-ca\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.453806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.453734 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.454049 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.453838 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.454811 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.454776 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda0c369-3d5b-47d2-a9e5-1962a873c20d-metrics-client-ca\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.456517 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.456494 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.456637 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.456590 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.456818 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.456801 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.457033 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.457015 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.457033 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.457008 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-thanos-querier-tls\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.457147 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.457031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda0c369-3d5b-47d2-a9e5-1962a873c20d-secret-grpc-tls\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.461604 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.461573 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6n2f\" (UniqueName: \"kubernetes.io/projected/cda0c369-3d5b-47d2-a9e5-1962a873c20d-kube-api-access-l6n2f\") pod \"thanos-querier-84864f89c5-7zbl8\" (UID: \"cda0c369-3d5b-47d2-a9e5-1962a873c20d\") " pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.545991 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.545949 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:15.705356 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:15.705161 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84864f89c5-7zbl8"] Aug 01 00:38:16.064827 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:16.064731 2568 generic.go:356] "Generic (PLEG): container finished" podID="d742d153-2e24-4cb2-9422-275facf5e706" containerID="c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b" exitCode=0 Aug 01 00:38:16.064827 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:16.064810 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerDied","Data":"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b"} Aug 01 00:38:16.067491 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:16.067461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v745r" event={"ID":"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68","Type":"ContainerStarted","Data":"0b62d9b23a9fd1f163707512ab6a6fccc01457927f636fcbb30533b7ecdf785b"} Aug 01 00:38:16.067621 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:16.067514 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v745r" event={"ID":"d25f9979-ce83-4d01-9b0b-aabb2ddc7c68","Type":"ContainerStarted","Data":"437b2834c2a5bb2d90f72bd933e55225eb28d80fd3cebe7e234053303e950057"} Aug 01 00:38:16.068927 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:16.068897 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" event={"ID":"cda0c369-3d5b-47d2-a9e5-1962a873c20d","Type":"ContainerStarted","Data":"2fbb9d6851cff583d410cd9bc87942e92465b95cb3cbd0f2a633a6244f4c69d6"} Aug 01 00:38:16.086099 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:16.086050 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v745r" podStartSLOduration=2.915726733 podStartE2EDuration="4.08603275s" totalImagesPullingTime="1.098476109s" totalInitContainerRuntime="71.829908ms" isStatefulPod=true podCreationTimestamp="2026-08-01 00:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:16.085171662 +0000 UTC m=+274.294800119" watchObservedRunningTime="2026-08-01 00:38:16.08603275 +0000 UTC m=+274.295661207" Aug 01 00:38:17.076717 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:17.076628 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerStarted","Data":"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a"} Aug 01 00:38:17.077130 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:17.076724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerStarted","Data":"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c"} Aug 01 00:38:17.077130 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:17.076766 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerStarted","Data":"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa"} Aug 01 00:38:17.077130 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:17.076811 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerStarted","Data":"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0"} Aug 01 00:38:17.077130 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:17.076850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerStarted","Data":"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78"} Aug 01 00:38:19.088698 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.088659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerStarted","Data":"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58"} Aug 01 00:38:19.091536 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.091513 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" event={"ID":"cda0c369-3d5b-47d2-a9e5-1962a873c20d","Type":"ContainerStarted","Data":"9d54260cafc7538c89c24853cd7cd9b1eb70e5d6d2463b834f01e8f3dc0c0fcc"} Aug 01 00:38:19.091644 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.091575 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" event={"ID":"cda0c369-3d5b-47d2-a9e5-1962a873c20d","Type":"ContainerStarted","Data":"20e61da2f323cea53261ac32824243157537d28eb0fd57dc6afef00530a2b044"} Aug 01 00:38:19.091644 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.091618 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" event={"ID":"cda0c369-3d5b-47d2-a9e5-1962a873c20d","Type":"ContainerStarted","Data":"4db60f76a8cac903053eb4ef800c4601d4d36d5dbd004b510aa5cbcb7402af83"} Aug 01 00:38:19.091711 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.091649 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" event={"ID":"cda0c369-3d5b-47d2-a9e5-1962a873c20d","Type":"ContainerStarted","Data":"6ced0638282abfde516e1feea81e532cd01efcbca93111b77a90524b40696d50"} Aug 01 00:38:19.091711 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.091679 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:19.091772 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.091722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" event={"ID":"cda0c369-3d5b-47d2-a9e5-1962a873c20d","Type":"ContainerStarted","Data":"11b16dad2d26a4533accd0139320454e99d88d5cacdf188f3cc75d60df4cde0c"} Aug 01 00:38:19.091772 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.091748 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" event={"ID":"cda0c369-3d5b-47d2-a9e5-1962a873c20d","Type":"ContainerStarted","Data":"269f3d25bd9edc42b009dd195c51f2a6fee1afe8bd80867b4280b24ab092c94f"} Aug 01 00:38:19.115998 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.115945 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.177799293 podStartE2EDuration="6.115931093s" totalImagesPullingTime="3.923792418s" totalInitContainerRuntime="14.339382ms" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:19.11453322 +0000 UTC m=+277.324161677" watchObservedRunningTime="2026-08-01 00:38:19.115931093 +0000 UTC m=+277.325559549" Aug 01 00:38:19.136871 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:19.136816 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" podStartSLOduration=1.68883117 podStartE2EDuration="4.136797598s" totalImagesPullingTime="2.447966428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:19.136393508 +0000 UTC m=+277.346021967" watchObservedRunningTime="2026-08-01 00:38:19.136797598 +0000 UTC m=+277.346426055" Aug 01 00:38:23.707890 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:38:23.707842 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" podUID="59dd5e67-b383-49a7-bae0-17338c424d9b" Aug 01 00:38:24.032593 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.032507 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-hzmfd"] Aug 01 00:38:24.036912 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.036890 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-hzmfd" Aug 01 00:38:24.039148 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.039110 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 01 00:38:24.039148 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.039140 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 01 00:38:24.039371 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.039248 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-m9j7b\"" Aug 01 00:38:24.046083 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.046058 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-hzmfd"] Aug 01 00:38:24.109996 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.109964 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:38:24.133190 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.133159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x997x\" (UniqueName: \"kubernetes.io/projected/4960ec19-0b91-4513-9a88-a0d078a45ec8-kube-api-access-x997x\") pod \"downloads-77c974ffcf-hzmfd\" (UID: \"4960ec19-0b91-4513-9a88-a0d078a45ec8\") " pod="openshift-console/downloads-77c974ffcf-hzmfd" Aug 01 00:38:24.234508 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.234478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x997x\" (UniqueName: \"kubernetes.io/projected/4960ec19-0b91-4513-9a88-a0d078a45ec8-kube-api-access-x997x\") pod \"downloads-77c974ffcf-hzmfd\" (UID: \"4960ec19-0b91-4513-9a88-a0d078a45ec8\") " pod="openshift-console/downloads-77c974ffcf-hzmfd" Aug 01 00:38:24.244157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.244131 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x997x\" (UniqueName: \"kubernetes.io/projected/4960ec19-0b91-4513-9a88-a0d078a45ec8-kube-api-access-x997x\") pod \"downloads-77c974ffcf-hzmfd\" (UID: \"4960ec19-0b91-4513-9a88-a0d078a45ec8\") " pod="openshift-console/downloads-77c974ffcf-hzmfd" Aug 01 00:38:24.346489 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.346384 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-hzmfd" Aug 01 00:38:24.481474 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:24.481329 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-hzmfd"] Aug 01 00:38:24.484171 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:38:24.484137 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4960ec19_0b91_4513_9a88_a0d078a45ec8.slice/crio-ec8511e16a1316f0b3d6ce3028cf4421594afcf0c06d5ee8c0daf6d16eb75c48 WatchSource:0}: Error finding container ec8511e16a1316f0b3d6ce3028cf4421594afcf0c06d5ee8c0daf6d16eb75c48: Status 404 returned error can't find the container with id ec8511e16a1316f0b3d6ce3028cf4421594afcf0c06d5ee8c0daf6d16eb75c48 Aug 01 00:38:25.102190 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:25.102163 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84864f89c5-7zbl8" Aug 01 00:38:25.116043 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:25.116010 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-hzmfd" event={"ID":"4960ec19-0b91-4513-9a88-a0d078a45ec8","Type":"ContainerStarted","Data":"ec8511e16a1316f0b3d6ce3028cf4421594afcf0c06d5ee8c0daf6d16eb75c48"} Aug 01 00:38:26.658870 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:26.658817 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:38:26.661676 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:26.661645 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"image-registry-67f6648b84-t5vxk\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:38:26.759556 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:26.759516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:38:26.762232 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:26.762194 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30615951-9c11-4b6b-9b0a-00813d2ec60e-metrics-tls\") pod \"dns-default-vpbm6\" (UID: \"30615951-9c11-4b6b-9b0a-00813d2ec60e\") " pod="openshift-dns/dns-default-vpbm6" Aug 01 00:38:26.813403 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:26.813365 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4zgh7\"" Aug 01 00:38:26.822095 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:26.822067 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:38:26.831883 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:26.831789 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qd5dz\"" Aug 01 00:38:26.837646 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:26.837625 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpbm6" Aug 01 00:38:27.000115 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:27.000079 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67f6648b84-t5vxk"] Aug 01 00:38:27.004367 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:38:27.004335 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dd5e67_b383_49a7_bae0_17338c424d9b.slice/crio-c438caebd4c7c258a4ca8a7f6f24e60c683a43a19d6e72b7ed3b26b6c9027b18 WatchSource:0}: Error finding container c438caebd4c7c258a4ca8a7f6f24e60c683a43a19d6e72b7ed3b26b6c9027b18: Status 404 returned error can't find the container with id c438caebd4c7c258a4ca8a7f6f24e60c683a43a19d6e72b7ed3b26b6c9027b18 Aug 01 00:38:27.017021 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:27.016952 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpbm6"] Aug 01 00:38:27.019833 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:38:27.019805 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30615951_9c11_4b6b_9b0a_00813d2ec60e.slice/crio-046b5094840b6b91af8c770ffbdfbd9ab74e6b313d3f44d70fa01b0fa5eb4a30 WatchSource:0}: Error finding container 046b5094840b6b91af8c770ffbdfbd9ab74e6b313d3f44d70fa01b0fa5eb4a30: Status 404 returned error can't find the container with id 046b5094840b6b91af8c770ffbdfbd9ab74e6b313d3f44d70fa01b0fa5eb4a30 Aug 01 00:38:27.125935 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:27.125895 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpbm6" event={"ID":"30615951-9c11-4b6b-9b0a-00813d2ec60e","Type":"ContainerStarted","Data":"046b5094840b6b91af8c770ffbdfbd9ab74e6b313d3f44d70fa01b0fa5eb4a30"} Aug 01 00:38:27.127711 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:27.127678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" event={"ID":"59dd5e67-b383-49a7-bae0-17338c424d9b","Type":"ContainerStarted","Data":"c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131"} Aug 01 00:38:27.127840 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:27.127724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" event={"ID":"59dd5e67-b383-49a7-bae0-17338c424d9b","Type":"ContainerStarted","Data":"c438caebd4c7c258a4ca8a7f6f24e60c683a43a19d6e72b7ed3b26b6c9027b18"} Aug 01 00:38:27.127952 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:27.127905 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:38:27.148894 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:27.148832 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" podStartSLOduration=285.148809195 podStartE2EDuration="4m45.148809195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:27.14768669 +0000 UTC m=+285.357315150" watchObservedRunningTime="2026-08-01 00:38:27.148809195 +0000 UTC m=+285.358437656" Aug 01 00:38:29.142342 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:29.142250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpbm6" event={"ID":"30615951-9c11-4b6b-9b0a-00813d2ec60e","Type":"ContainerStarted","Data":"3f4888ed74d4e97cbdd26d411129eeb66be6871a53325f82f43c4dbebc997946"} Aug 01 00:38:29.142342 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:29.142315 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpbm6" event={"ID":"30615951-9c11-4b6b-9b0a-00813d2ec60e","Type":"ContainerStarted","Data":"70f2afd06e629fc636b6fd3c54ce71569994d522890083002b8cd911a4ee3528"} Aug 01 00:38:29.142772 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:29.142441 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vpbm6" Aug 01 00:38:29.157703 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:29.157649 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vpbm6" podStartSLOduration=251.336683426 podStartE2EDuration="4m13.157632142s" totalImagesPullingTime="1.820948716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:29.157538743 +0000 UTC m=+287.367167202" watchObservedRunningTime="2026-08-01 00:38:29.157632142 +0000 UTC m=+287.367260599" Aug 01 00:38:34.558509 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.558475 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77d4459d9d-2xqfr"] Aug 01 00:38:34.562733 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.562713 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.566497 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.566465 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 01 00:38:34.566497 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.566483 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8tgrd\"" Aug 01 00:38:34.566689 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.566465 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 01 00:38:34.566768 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.566747 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 01 00:38:34.566768 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.566758 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 01 00:38:34.566966 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.566808 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 01 00:38:34.573537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.573334 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d4459d9d-2xqfr"] Aug 01 00:38:34.630225 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.630192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-oauth-config\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.630390 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.630255 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-config\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.630390 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.630318 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjh6\" (UniqueName: \"kubernetes.io/projected/bee5a83e-34cd-4aef-842f-f0826cf7544c-kube-api-access-mdjh6\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.630390 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.630353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-service-ca\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.630570 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.630461 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-oauth-serving-cert\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.630570 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.630517 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-serving-cert\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.731649 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.731607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-oauth-serving-cert\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.731816 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.731676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-serving-cert\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.731816 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.731715 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-oauth-config\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.731816 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.731739 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-config\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.731955 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.731815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjh6\" (UniqueName: \"kubernetes.io/projected/bee5a83e-34cd-4aef-842f-f0826cf7544c-kube-api-access-mdjh6\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.731955 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.731870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-service-ca\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.732500 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.732474 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-oauth-serving-cert\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.732633 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.732480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-config\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.732633 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.732611 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-service-ca\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.734692 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.734655 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-oauth-config\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.734806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.734780 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-serving-cert\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.740154 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.740129 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjh6\" (UniqueName: \"kubernetes.io/projected/bee5a83e-34cd-4aef-842f-f0826cf7544c-kube-api-access-mdjh6\") pod \"console-77d4459d9d-2xqfr\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:34.873766 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:34.873674 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:36.224918 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:38:36.224878 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-249m9" podUID="dcf38574-98e0-464c-8f21-bdb577e4b0ec" Aug 01 00:38:39.149617 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:39.149580 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vpbm6" Aug 01 00:38:40.901732 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:40.901696 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d4459d9d-2xqfr"] Aug 01 00:38:40.918328 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:38:40.918297 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee5a83e_34cd_4aef_842f_f0826cf7544c.slice/crio-7eff1cf09026af63190f03aae7bde6be6d5ed2d12a30309a2ea0a7c54721dd52 WatchSource:0}: Error finding container 7eff1cf09026af63190f03aae7bde6be6d5ed2d12a30309a2ea0a7c54721dd52: Status 404 returned error can't find the container with id 7eff1cf09026af63190f03aae7bde6be6d5ed2d12a30309a2ea0a7c54721dd52 Aug 01 00:38:41.199044 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:41.198974 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4459d9d-2xqfr" event={"ID":"bee5a83e-34cd-4aef-842f-f0826cf7544c","Type":"ContainerStarted","Data":"7eff1cf09026af63190f03aae7bde6be6d5ed2d12a30309a2ea0a7c54721dd52"} Aug 01 00:38:41.200868 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:41.200839 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-hzmfd" event={"ID":"4960ec19-0b91-4513-9a88-a0d078a45ec8","Type":"ContainerStarted","Data":"55d4064d0f2537c20ef41de6325bd3383f1a00f09f6b471be82f23dd6fa3c8a8"} Aug 01 00:38:41.201096 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:41.201052 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-hzmfd" Aug 01 00:38:41.213463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:41.213435 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-hzmfd" Aug 01 00:38:41.218954 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:41.218905 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-hzmfd" podStartSLOduration=0.827227071 podStartE2EDuration="17.218888885s" totalImagesPullingTime="16.391661814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:41.216974647 +0000 UTC m=+299.426603103" watchObservedRunningTime="2026-08-01 00:38:41.218888885 +0000 UTC m=+299.428517341" Aug 01 00:38:42.185235 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:42.185175 2568 kubelet.go:1648] "Image garbage collection succeeded" Aug 01 00:38:45.226442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:45.226389 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4459d9d-2xqfr" event={"ID":"bee5a83e-34cd-4aef-842f-f0826cf7544c","Type":"ContainerStarted","Data":"ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d"} Aug 01 00:38:45.244089 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:45.244024 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77d4459d9d-2xqfr" podStartSLOduration=7.516876734 podStartE2EDuration="11.244003387s" totalImagesPullingTime="3.727126653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:45.243422893 +0000 UTC m=+303.453051350" watchObservedRunningTime="2026-08-01 00:38:45.244003387 +0000 UTC m=+303.453631848" Aug 01 00:38:46.827600 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:46.827543 2568 patch_prober.go:28] interesting pod/image-registry-67f6648b84-t5vxk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 01 00:38:46.828098 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:46.827631 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" podUID="59dd5e67-b383-49a7-bae0-17338c424d9b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 00:38:48.138017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:48.137984 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:38:51.224926 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:51.224875 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:38:54.874822 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:54.874787 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:54.875304 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:54.874849 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:54.879729 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:54.879710 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:55.235119 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:55.235085 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67f6648b84-t5vxk"] Aug 01 00:38:55.270674 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:55.270646 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:38:55.273858 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:38:55.273835 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d4459d9d-2xqfr"] Aug 01 00:39:13.335945 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:13.335863 2568 generic.go:356] "Generic (PLEG): container finished" podID="cac70d06-1990-4dbf-83ee-f221288845aa" containerID="81a5cdaa46ef9356a822b6f8a61c11d666321df7ee54cfa288f7ea502cc28ee0" exitCode=0 Aug 01 00:39:13.335945 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:13.335907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" event={"ID":"cac70d06-1990-4dbf-83ee-f221288845aa","Type":"ContainerDied","Data":"81a5cdaa46ef9356a822b6f8a61c11d666321df7ee54cfa288f7ea502cc28ee0"} Aug 01 00:39:13.336389 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:13.336239 2568 scope.go:117] "RemoveContainer" containerID="81a5cdaa46ef9356a822b6f8a61c11d666321df7ee54cfa288f7ea502cc28ee0" Aug 01 00:39:14.342004 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:14.341962 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-8m8rz" event={"ID":"cac70d06-1990-4dbf-83ee-f221288845aa","Type":"ContainerStarted","Data":"86780ae2ca8b432335a90b653aafa3d44b318d2337f0622def76048160f0e6f6"} Aug 01 00:39:14.343495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:14.343464 2568 generic.go:356] "Generic (PLEG): container finished" podID="7e2afe34-c831-4376-a249-119e5eb68763" containerID="b07322ce8723924c267dc07d047ebd542d70cea13a9eb7fb5a28f75b4bc38879" exitCode=0 Aug 01 00:39:14.343628 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:14.343545 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-zmdst" event={"ID":"7e2afe34-c831-4376-a249-119e5eb68763","Type":"ContainerDied","Data":"b07322ce8723924c267dc07d047ebd542d70cea13a9eb7fb5a28f75b4bc38879"} Aug 01 00:39:14.343899 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:14.343883 2568 scope.go:117] "RemoveContainer" containerID="b07322ce8723924c267dc07d047ebd542d70cea13a9eb7fb5a28f75b4bc38879" Aug 01 00:39:15.348862 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:15.348827 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-zmdst" event={"ID":"7e2afe34-c831-4376-a249-119e5eb68763","Type":"ContainerStarted","Data":"560a5fdddae9dcb2d200d9492d9b744e21d5a2b0e8bf5d29a0ba92eb69b5a268"} Aug 01 00:39:15.383276 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:15.383249 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67f6648b84-t5vxk_59dd5e67-b383-49a7-bae0-17338c424d9b/registry/0.log" Aug 01 00:39:15.980424 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:15.980399 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vm2k4_06e31806-756d-4782-9193-ac8ce656d17b/node-ca/0.log" Aug 01 00:39:16.179735 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:16.179699 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64956b7558-rqmzr_5027f8b0-fdc9-48e1-bdd1-898dafd7fae2/router/0.log" Aug 01 00:39:16.780041 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:16.780012 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kvg6c_8cdf8609-6802-4dc9-b5b6-ca974534987b/serve-healthcheck-canary/0.log" Aug 01 00:39:18.361524 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:18.361441 2568 generic.go:356] "Generic (PLEG): container finished" podID="f7b5f441-b04a-4b1c-9b75-9bf091c54a21" containerID="2fc03602624202dbe97d30839319ad6d3e129ba176e8b392382d24ab0913ac36" exitCode=0 Aug 01 00:39:18.361876 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:18.361516 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" event={"ID":"f7b5f441-b04a-4b1c-9b75-9bf091c54a21","Type":"ContainerDied","Data":"2fc03602624202dbe97d30839319ad6d3e129ba176e8b392382d24ab0913ac36"} Aug 01 00:39:18.361876 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:18.361814 2568 scope.go:117] "RemoveContainer" containerID="2fc03602624202dbe97d30839319ad6d3e129ba176e8b392382d24ab0913ac36" Aug 01 00:39:19.369626 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:19.369592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6gx8p" event={"ID":"f7b5f441-b04a-4b1c-9b75-9bf091c54a21","Type":"ContainerStarted","Data":"efb668d45188d6fd70dcccf3ef95750fc744a7fb81dbe57bf2b2e8701a9e9ed9"} Aug 01 00:39:20.254452 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:20.254406 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" podUID="59dd5e67-b383-49a7-bae0-17338c424d9b" containerName="registry" containerID="cri-o://c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131" gracePeriod=30 Aug 01 00:39:22.021094 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.021067 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:39:22.159779 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.159702 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-certificates\") pod \"59dd5e67-b383-49a7-bae0-17338c424d9b\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " Aug 01 00:39:22.159779 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.159736 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-installation-pull-secrets\") pod \"59dd5e67-b383-49a7-bae0-17338c424d9b\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " Aug 01 00:39:22.159779 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.159773 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-bound-sa-token\") pod \"59dd5e67-b383-49a7-bae0-17338c424d9b\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " Aug 01 00:39:22.160052 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.159792 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-image-registry-private-configuration\") pod \"59dd5e67-b383-49a7-bae0-17338c424d9b\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " Aug 01 00:39:22.160052 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.159821 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmxrx\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-kube-api-access-jmxrx\") pod \"59dd5e67-b383-49a7-bae0-17338c424d9b\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " Aug 01 00:39:22.160052 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.159841 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59dd5e67-b383-49a7-bae0-17338c424d9b-ca-trust-extracted\") pod \"59dd5e67-b383-49a7-bae0-17338c424d9b\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " Aug 01 00:39:22.160052 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.159858 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") pod \"59dd5e67-b383-49a7-bae0-17338c424d9b\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " Aug 01 00:39:22.160052 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.159909 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-trusted-ca\") pod \"59dd5e67-b383-49a7-bae0-17338c424d9b\" (UID: \"59dd5e67-b383-49a7-bae0-17338c424d9b\") " Aug 01 00:39:22.160563 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.160476 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "59dd5e67-b383-49a7-bae0-17338c424d9b" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:39:22.160563 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.160523 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "59dd5e67-b383-49a7-bae0-17338c424d9b" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:39:22.162524 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.162493 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "59dd5e67-b383-49a7-bae0-17338c424d9b" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:22.162636 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.162488 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-kube-api-access-jmxrx" (OuterVolumeSpecName: "kube-api-access-jmxrx") pod "59dd5e67-b383-49a7-bae0-17338c424d9b" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b"). InnerVolumeSpecName "kube-api-access-jmxrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:39:22.162636 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.162545 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "59dd5e67-b383-49a7-bae0-17338c424d9b" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:39:22.162945 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.162914 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "59dd5e67-b383-49a7-bae0-17338c424d9b" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:22.162945 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.162935 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "59dd5e67-b383-49a7-bae0-17338c424d9b" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:39:22.169341 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.169312 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dd5e67-b383-49a7-bae0-17338c424d9b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "59dd5e67-b383-49a7-bae0-17338c424d9b" (UID: "59dd5e67-b383-49a7-bae0-17338c424d9b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:39:22.260572 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.260536 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-certificates\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.260572 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.260563 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-installation-pull-secrets\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.260728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.260577 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-bound-sa-token\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.260728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.260591 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59dd5e67-b383-49a7-bae0-17338c424d9b-image-registry-private-configuration\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.260728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.260604 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxrx\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-kube-api-access-jmxrx\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.260728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.260615 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59dd5e67-b383-49a7-bae0-17338c424d9b-ca-trust-extracted\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.260728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.260627 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59dd5e67-b383-49a7-bae0-17338c424d9b-registry-tls\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.260728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.260637 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59dd5e67-b383-49a7-bae0-17338c424d9b-trusted-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.294151 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.294117 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77d4459d9d-2xqfr" podUID="bee5a83e-34cd-4aef-842f-f0826cf7544c" containerName="console" containerID="cri-o://ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d" gracePeriod=15 Aug 01 00:39:22.381689 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.381660 2568 generic.go:356] "Generic (PLEG): container finished" podID="59dd5e67-b383-49a7-bae0-17338c424d9b" containerID="c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131" exitCode=0 Aug 01 00:39:22.381796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.381705 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" Aug 01 00:39:22.381796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.381724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" event={"ID":"59dd5e67-b383-49a7-bae0-17338c424d9b","Type":"ContainerDied","Data":"c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131"} Aug 01 00:39:22.381796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.381763 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67f6648b84-t5vxk" event={"ID":"59dd5e67-b383-49a7-bae0-17338c424d9b","Type":"ContainerDied","Data":"c438caebd4c7c258a4ca8a7f6f24e60c683a43a19d6e72b7ed3b26b6c9027b18"} Aug 01 00:39:22.381796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.381792 2568 scope.go:117] "RemoveContainer" containerID="c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131" Aug 01 00:39:22.398491 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.398397 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67f6648b84-t5vxk"] Aug 01 00:39:22.402486 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.402466 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67f6648b84-t5vxk"] Aug 01 00:39:22.405312 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.405287 2568 scope.go:117] "RemoveContainer" containerID="c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131" Aug 01 00:39:22.405628 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:22.405598 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131\": container with ID starting with c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131 not found: ID does not exist" containerID="c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131" Aug 01 00:39:22.405719 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.405636 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131"} err="failed to get container status \"c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131\": rpc error: code = NotFound desc = could not find container \"c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131\": container with ID starting with c5ef839456e4ce553a1c239ae455ce795085d27812c2f106f3f68cc5ba802131 not found: ID does not exist" Aug 01 00:39:22.575411 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.575387 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:39:22.664914 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.664885 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-config\") pod \"bee5a83e-34cd-4aef-842f-f0826cf7544c\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " Aug 01 00:39:22.665080 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.664926 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdjh6\" (UniqueName: \"kubernetes.io/projected/bee5a83e-34cd-4aef-842f-f0826cf7544c-kube-api-access-mdjh6\") pod \"bee5a83e-34cd-4aef-842f-f0826cf7544c\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " Aug 01 00:39:22.665080 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.664944 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-service-ca\") pod \"bee5a83e-34cd-4aef-842f-f0826cf7544c\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " Aug 01 00:39:22.665080 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.664967 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-oauth-config\") pod \"bee5a83e-34cd-4aef-842f-f0826cf7544c\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " Aug 01 00:39:22.665080 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.664987 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-serving-cert\") pod \"bee5a83e-34cd-4aef-842f-f0826cf7544c\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " Aug 01 00:39:22.665302 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.665172 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-oauth-serving-cert\") pod \"bee5a83e-34cd-4aef-842f-f0826cf7544c\" (UID: \"bee5a83e-34cd-4aef-842f-f0826cf7544c\") " Aug 01 00:39:22.665369 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.665344 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-config" (OuterVolumeSpecName: "console-config") pod "bee5a83e-34cd-4aef-842f-f0826cf7544c" (UID: "bee5a83e-34cd-4aef-842f-f0826cf7544c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:39:22.665430 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.665375 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-service-ca" (OuterVolumeSpecName: "service-ca") pod "bee5a83e-34cd-4aef-842f-f0826cf7544c" (UID: "bee5a83e-34cd-4aef-842f-f0826cf7544c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:39:22.665540 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.665517 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bee5a83e-34cd-4aef-842f-f0826cf7544c" (UID: "bee5a83e-34cd-4aef-842f-f0826cf7544c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:39:22.665599 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.665576 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-config\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.665599 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.665590 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.667553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.667528 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee5a83e-34cd-4aef-842f-f0826cf7544c-kube-api-access-mdjh6" (OuterVolumeSpecName: "kube-api-access-mdjh6") pod "bee5a83e-34cd-4aef-842f-f0826cf7544c" (UID: "bee5a83e-34cd-4aef-842f-f0826cf7544c"). InnerVolumeSpecName "kube-api-access-mdjh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:39:22.667553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.667539 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bee5a83e-34cd-4aef-842f-f0826cf7544c" (UID: "bee5a83e-34cd-4aef-842f-f0826cf7544c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:22.667697 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.667605 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bee5a83e-34cd-4aef-842f-f0826cf7544c" (UID: "bee5a83e-34cd-4aef-842f-f0826cf7544c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:22.766855 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.766787 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdjh6\" (UniqueName: \"kubernetes.io/projected/bee5a83e-34cd-4aef-842f-f0826cf7544c-kube-api-access-mdjh6\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.766855 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.766812 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-oauth-config\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.766855 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.766822 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bee5a83e-34cd-4aef-842f-f0826cf7544c-console-serving-cert\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:22.766855 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:22.766833 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bee5a83e-34cd-4aef-842f-f0826cf7544c-oauth-serving-cert\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:23.387808 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.387776 2568 generic.go:356] "Generic (PLEG): container finished" podID="bee5a83e-34cd-4aef-842f-f0826cf7544c" containerID="ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d" exitCode=0 Aug 01 00:39:23.388203 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.387819 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4459d9d-2xqfr" Aug 01 00:39:23.388203 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.387833 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4459d9d-2xqfr" event={"ID":"bee5a83e-34cd-4aef-842f-f0826cf7544c","Type":"ContainerDied","Data":"ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d"} Aug 01 00:39:23.388203 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.387870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4459d9d-2xqfr" event={"ID":"bee5a83e-34cd-4aef-842f-f0826cf7544c","Type":"ContainerDied","Data":"7eff1cf09026af63190f03aae7bde6be6d5ed2d12a30309a2ea0a7c54721dd52"} Aug 01 00:39:23.388203 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.387890 2568 scope.go:117] "RemoveContainer" containerID="ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d" Aug 01 00:39:23.396835 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.396817 2568 scope.go:117] "RemoveContainer" containerID="ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d" Aug 01 00:39:23.397077 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:23.397058 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d\": container with ID starting with ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d not found: ID does not exist" containerID="ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d" Aug 01 00:39:23.397137 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.397086 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d"} err="failed to get container status \"ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d\": rpc error: code = NotFound desc = could not find container \"ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d\": container with ID starting with ba00d526e178df61e98577df678b0af27efdcd69829d6feffc8b880896cae97d not found: ID does not exist" Aug 01 00:39:23.407737 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.407715 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d4459d9d-2xqfr"] Aug 01 00:39:23.413336 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:23.413319 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77d4459d9d-2xqfr"] Aug 01 00:39:24.229481 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:24.229451 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59dd5e67-b383-49a7-bae0-17338c424d9b" path="/var/lib/kubelet/pods/59dd5e67-b383-49a7-bae0-17338c424d9b/volumes" Aug 01 00:39:24.229834 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:24.229821 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee5a83e-34cd-4aef-842f-f0826cf7544c" path="/var/lib/kubelet/pods/bee5a83e-34cd-4aef-842f-f0826cf7544c/volumes" Aug 01 00:39:32.476603 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:32.476571 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:39:32.477017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:32.476836 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="alertmanager" containerID="cri-o://f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78" gracePeriod=120 Aug 01 00:39:32.477017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:32.476921 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy" containerID="cri-o://7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c" gracePeriod=120 Aug 01 00:39:32.477017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:32.476939 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy-web" containerID="cri-o://20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa" gracePeriod=120 Aug 01 00:39:32.477017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:32.476947 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="config-reloader" containerID="cri-o://549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0" gracePeriod=120 Aug 01 00:39:32.477017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:32.476917 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy-metric" containerID="cri-o://645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a" gracePeriod=120 Aug 01 00:39:32.477303 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:32.477002 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="prom-label-proxy" containerID="cri-o://f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58" gracePeriod=120 Aug 01 00:39:33.433039 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.433006 2568 generic.go:356] "Generic (PLEG): container finished" podID="d742d153-2e24-4cb2-9422-275facf5e706" containerID="f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58" exitCode=0 Aug 01 00:39:33.433039 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.433030 2568 generic.go:356] "Generic (PLEG): container finished" podID="d742d153-2e24-4cb2-9422-275facf5e706" containerID="7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c" exitCode=0 Aug 01 00:39:33.433039 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.433036 2568 generic.go:356] "Generic (PLEG): container finished" podID="d742d153-2e24-4cb2-9422-275facf5e706" containerID="549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0" exitCode=0 Aug 01 00:39:33.433039 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.433041 2568 generic.go:356] "Generic (PLEG): container finished" podID="d742d153-2e24-4cb2-9422-275facf5e706" containerID="f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78" exitCode=0 Aug 01 00:39:33.433384 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.433079 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerDied","Data":"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58"} Aug 01 00:39:33.433384 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.433191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerDied","Data":"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c"} Aug 01 00:39:33.433384 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.433248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerDied","Data":"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0"} Aug 01 00:39:33.433384 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.433286 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerDied","Data":"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78"} Aug 01 00:39:33.715582 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.715558 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:33.863027 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.862988 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-metrics-client-ca\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863027 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863032 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-config-volume\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863249 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863050 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863249 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863074 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-main-tls\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863342 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863282 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-cluster-tls-config\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863342 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863329 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-main-db\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863443 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863386 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-trusted-ca-bundle\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863443 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863402 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:39:33.863443 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863415 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-web-config\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863611 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863494 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-web\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863611 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863546 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-tls-assets\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863611 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863605 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d94pn\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-kube-api-access-d94pn\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863753 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863642 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-config-out\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863753 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863668 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy\") pod \"d742d153-2e24-4cb2-9422-275facf5e706\" (UID: \"d742d153-2e24-4cb2-9422-275facf5e706\") " Aug 01 00:39:33.863912 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.863866 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:39:33.864438 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.864115 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-main-db\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.864438 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.864142 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-metrics-client-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.864438 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.864237 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:39:33.866259 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.866206 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:33.866748 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.866712 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-config-volume" (OuterVolumeSpecName: "config-volume") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:33.866985 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.866953 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-config-out" (OuterVolumeSpecName: "config-out") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:39:33.867088 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.867000 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:33.867206 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.867184 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:33.867736 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.867708 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:33.867847 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.867800 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-kube-api-access-d94pn" (OuterVolumeSpecName: "kube-api-access-d94pn") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "kube-api-access-d94pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:39:33.868012 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.867993 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:39:33.871636 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.871608 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:33.877017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.876997 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-web-config" (OuterVolumeSpecName: "web-config") pod "d742d153-2e24-4cb2-9422-275facf5e706" (UID: "d742d153-2e24-4cb2-9422-275facf5e706"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:39:33.964758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964675 2568 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.964758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964708 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.964758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964719 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-main-tls\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.964758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964728 2568 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-cluster-tls-config\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.964758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964737 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d742d153-2e24-4cb2-9422-275facf5e706-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.964758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964746 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-web-config\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.964758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964754 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.964758 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964762 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-tls-assets\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.965079 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964771 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94pn\" (UniqueName: \"kubernetes.io/projected/d742d153-2e24-4cb2-9422-275facf5e706-kube-api-access-d94pn\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.965079 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964778 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d742d153-2e24-4cb2-9422-275facf5e706-config-out\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:33.965079 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:33.964786 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d742d153-2e24-4cb2-9422-275facf5e706-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:39:34.440569 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.440531 2568 generic.go:356] "Generic (PLEG): container finished" podID="d742d153-2e24-4cb2-9422-275facf5e706" containerID="645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a" exitCode=0 Aug 01 00:39:34.440569 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.440560 2568 generic.go:356] "Generic (PLEG): container finished" podID="d742d153-2e24-4cb2-9422-275facf5e706" containerID="20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa" exitCode=0 Aug 01 00:39:34.440789 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.440573 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerDied","Data":"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a"} Aug 01 00:39:34.440789 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.440610 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.440789 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.440644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerDied","Data":"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa"} Aug 01 00:39:34.440789 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.440671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d742d153-2e24-4cb2-9422-275facf5e706","Type":"ContainerDied","Data":"71799f90435e3e6a2f1334bc46131b1afe849015f5096181b8cec3407a770ece"} Aug 01 00:39:34.440789 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.440701 2568 scope.go:117] "RemoveContainer" containerID="f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58" Aug 01 00:39:34.449776 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.449749 2568 scope.go:117] "RemoveContainer" containerID="645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a" Aug 01 00:39:34.456855 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.456837 2568 scope.go:117] "RemoveContainer" containerID="7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c" Aug 01 00:39:34.459576 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.459553 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:39:34.463888 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.463864 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:39:34.464468 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.464454 2568 scope.go:117] "RemoveContainer" containerID="20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa" Aug 01 00:39:34.471458 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.471440 2568 scope.go:117] "RemoveContainer" containerID="549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0" Aug 01 00:39:34.478340 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.478325 2568 scope.go:117] "RemoveContainer" containerID="f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78" Aug 01 00:39:34.485341 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.485326 2568 scope.go:117] "RemoveContainer" containerID="c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b" Aug 01 00:39:34.491955 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.491900 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:39:34.492191 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492175 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee5a83e-34cd-4aef-842f-f0826cf7544c" containerName="console" Aug 01 00:39:34.492262 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492195 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee5a83e-34cd-4aef-842f-f0826cf7544c" containerName="console" Aug 01 00:39:34.492262 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492207 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="config-reloader" Aug 01 00:39:34.492262 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492233 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="config-reloader" Aug 01 00:39:34.492262 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492249 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59dd5e67-b383-49a7-bae0-17338c424d9b" containerName="registry" Aug 01 00:39:34.492262 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492257 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59dd5e67-b383-49a7-bae0-17338c424d9b" containerName="registry" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492260 2568 scope.go:117] "RemoveContainer" containerID="f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492268 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="prom-label-proxy" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492314 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="prom-label-proxy" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492337 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy-web" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492342 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy-web" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492351 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="init-config-reloader" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492356 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="init-config-reloader" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492364 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492370 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492375 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy-metric" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492381 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy-metric" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492388 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="alertmanager" Aug 01 00:39:34.492442 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492393 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="alertmanager" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492473 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492485 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee5a83e-34cd-4aef-842f-f0826cf7544c" containerName="console" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492492 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="config-reloader" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492498 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="alertmanager" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492503 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy-web" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492517 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="kube-rbac-proxy-metric" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492526 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="59dd5e67-b383-49a7-bae0-17338c424d9b" containerName="registry" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492531 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d742d153-2e24-4cb2-9422-275facf5e706" containerName="prom-label-proxy" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:34.492499 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58\": container with ID starting with f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58 not found: ID does not exist" containerID="f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492605 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58"} err="failed to get container status \"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58\": rpc error: code = NotFound desc = could not find container \"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58\": container with ID starting with f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58 not found: ID does not exist" Aug 01 00:39:34.492863 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492628 2568 scope.go:117] "RemoveContainer" containerID="645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a" Aug 01 00:39:34.493193 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:34.492879 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a\": container with ID starting with 645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a not found: ID does not exist" containerID="645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a" Aug 01 00:39:34.493193 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492898 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a"} err="failed to get container status \"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a\": rpc error: code = NotFound desc = could not find container \"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a\": container with ID starting with 645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a not found: ID does not exist" Aug 01 00:39:34.493193 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.492913 2568 scope.go:117] "RemoveContainer" containerID="7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c" Aug 01 00:39:34.493193 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:34.493141 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c\": container with ID starting with 7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c not found: ID does not exist" containerID="7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c" Aug 01 00:39:34.493193 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.493167 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c"} err="failed to get container status \"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c\": rpc error: code = NotFound desc = could not find container \"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c\": container with ID starting with 7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c not found: ID does not exist" Aug 01 00:39:34.493193 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.493185 2568 scope.go:117] "RemoveContainer" containerID="20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa" Aug 01 00:39:34.493456 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:34.493423 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa\": container with ID starting with 20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa not found: ID does not exist" containerID="20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa" Aug 01 00:39:34.493456 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.493441 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa"} err="failed to get container status \"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa\": rpc error: code = NotFound desc = could not find container \"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa\": container with ID starting with 20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa not found: ID does not exist" Aug 01 00:39:34.493520 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.493464 2568 scope.go:117] "RemoveContainer" containerID="549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0" Aug 01 00:39:34.493698 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:34.493678 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0\": container with ID starting with 549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0 not found: ID does not exist" containerID="549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0" Aug 01 00:39:34.493748 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.493704 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0"} err="failed to get container status \"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0\": rpc error: code = NotFound desc = could not find container \"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0\": container with ID starting with 549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0 not found: ID does not exist" Aug 01 00:39:34.493748 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.493723 2568 scope.go:117] "RemoveContainer" containerID="f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78" Aug 01 00:39:34.493967 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:34.493943 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78\": container with ID starting with f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78 not found: ID does not exist" containerID="f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78" Aug 01 00:39:34.494047 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.493977 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78"} err="failed to get container status \"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78\": rpc error: code = NotFound desc = could not find container \"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78\": container with ID starting with f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78 not found: ID does not exist" Aug 01 00:39:34.494047 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.493993 2568 scope.go:117] "RemoveContainer" containerID="c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b" Aug 01 00:39:34.494225 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:39:34.494196 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b\": container with ID starting with c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b not found: ID does not exist" containerID="c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b" Aug 01 00:39:34.494271 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.494229 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b"} err="failed to get container status \"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b\": rpc error: code = NotFound desc = could not find container \"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b\": container with ID starting with c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b not found: ID does not exist" Aug 01 00:39:34.494271 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.494243 2568 scope.go:117] "RemoveContainer" containerID="f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58" Aug 01 00:39:34.494458 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.494435 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58"} err="failed to get container status \"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58\": rpc error: code = NotFound desc = could not find container \"f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58\": container with ID starting with f5b43bf50809bba60d2ef81518be47ab8a70ff7cbe14fd013efda2281fb8ed58 not found: ID does not exist" Aug 01 00:39:34.494513 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.494460 2568 scope.go:117] "RemoveContainer" containerID="645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a" Aug 01 00:39:34.494694 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.494679 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a"} err="failed to get container status \"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a\": rpc error: code = NotFound desc = could not find container \"645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a\": container with ID starting with 645942a286716f3b4a1598bf44f75dfc728024ae605aa9b94617cb53035e884a not found: ID does not exist" Aug 01 00:39:34.494731 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.494695 2568 scope.go:117] "RemoveContainer" containerID="7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c" Aug 01 00:39:34.494931 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.494914 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c"} err="failed to get container status \"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c\": rpc error: code = NotFound desc = could not find container \"7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c\": container with ID starting with 7a0d4a798ef3df1ce3696676eb5ada8838f8dfef1ecdf1bd3470867e3d2a008c not found: ID does not exist" Aug 01 00:39:34.494980 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.494931 2568 scope.go:117] "RemoveContainer" containerID="20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa" Aug 01 00:39:34.495142 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.495124 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa"} err="failed to get container status \"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa\": rpc error: code = NotFound desc = could not find container \"20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa\": container with ID starting with 20852c075ffb8f00c390f4ca0f05fee706b8c7c19d73be1b27d474977bf9d5fa not found: ID does not exist" Aug 01 00:39:34.495182 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.495142 2568 scope.go:117] "RemoveContainer" containerID="549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0" Aug 01 00:39:34.495378 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.495362 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0"} err="failed to get container status \"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0\": rpc error: code = NotFound desc = could not find container \"549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0\": container with ID starting with 549f3931e7a8f934fb38a28dd38c7b7f967bcd66b7a99e1a6d0e938f3e4df4a0 not found: ID does not exist" Aug 01 00:39:34.495378 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.495378 2568 scope.go:117] "RemoveContainer" containerID="f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78" Aug 01 00:39:34.495552 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.495538 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78"} err="failed to get container status \"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78\": rpc error: code = NotFound desc = could not find container \"f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78\": container with ID starting with f0e65433275641daaf0bd2e37ed750ac7d56c3caf416ba12589d7016c4f88a78 not found: ID does not exist" Aug 01 00:39:34.495591 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.495552 2568 scope.go:117] "RemoveContainer" containerID="c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b" Aug 01 00:39:34.495743 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.495730 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b"} err="failed to get container status \"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b\": rpc error: code = NotFound desc = could not find container \"c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b\": container with ID starting with c24326998f6d40a80e66a38d75217d8d727f7940f73ceb29922c4f836b15206b not found: ID does not exist" Aug 01 00:39:34.496235 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.496190 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.498384 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.498367 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 01 00:39:34.498483 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.498404 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 01 00:39:34.498483 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.498417 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 01 00:39:34.498644 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.498624 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 01 00:39:34.498847 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.498830 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 01 00:39:34.498920 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.498836 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zrnxp\"" Aug 01 00:39:34.498920 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.498875 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 01 00:39:34.499025 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.498954 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 01 00:39:34.499179 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.499164 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 01 00:39:34.503877 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.503856 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 01 00:39:34.509980 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.509962 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:39:34.570476 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cde15bfa-85a5-4c51-b671-a52e95534f8f-config-out\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570476 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570474 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cde15bfa-85a5-4c51-b671-a52e95534f8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570681 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570496 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570681 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570681 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570646 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cde15bfa-85a5-4c51-b671-a52e95534f8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570782 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570691 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570782 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-web-config\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570782 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4bmc\" (UniqueName: \"kubernetes.io/projected/cde15bfa-85a5-4c51-b671-a52e95534f8f-kube-api-access-h4bmc\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570884 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570884 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde15bfa-85a5-4c51-b671-a52e95534f8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570884 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570865 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570986 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.570986 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.570905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cde15bfa-85a5-4c51-b671-a52e95534f8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.671744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.671708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.671744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.671745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672056 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.671766 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cde15bfa-85a5-4c51-b671-a52e95534f8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672056 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.671966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cde15bfa-85a5-4c51-b671-a52e95534f8f-config-out\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672056 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cde15bfa-85a5-4c51-b671-a52e95534f8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672239 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672058 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672239 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672239 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cde15bfa-85a5-4c51-b671-a52e95534f8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672239 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672239 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672181 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-web-config\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672239 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4bmc\" (UniqueName: \"kubernetes.io/projected/cde15bfa-85a5-4c51-b671-a52e95534f8f-kube-api-access-h4bmc\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672528 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672528 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde15bfa-85a5-4c51-b671-a52e95534f8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.672741 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.672671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cde15bfa-85a5-4c51-b671-a52e95534f8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.673197 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.673121 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cde15bfa-85a5-4c51-b671-a52e95534f8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.673611 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.673586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde15bfa-85a5-4c51-b671-a52e95534f8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.674928 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.674888 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cde15bfa-85a5-4c51-b671-a52e95534f8f-config-out\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.675194 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.675176 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.675488 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.675467 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.675542 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.675501 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.675939 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.675920 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.676062 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.676042 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.676148 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.676126 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cde15bfa-85a5-4c51-b671-a52e95534f8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.677000 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.676971 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-web-config\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.677335 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.677318 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cde15bfa-85a5-4c51-b671-a52e95534f8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.680628 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.680608 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4bmc\" (UniqueName: \"kubernetes.io/projected/cde15bfa-85a5-4c51-b671-a52e95534f8f-kube-api-access-h4bmc\") pod \"alertmanager-main-0\" (UID: \"cde15bfa-85a5-4c51-b671-a52e95534f8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.806060 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.805964 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 01 00:39:34.941914 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:34.941883 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 01 00:39:34.945673 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:39:34.945648 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde15bfa_85a5_4c51_b671_a52e95534f8f.slice/crio-56e90c8efbe401e94636f6afd5b4c7bf747d5a2a6491f0c586b68699a8b7393a WatchSource:0}: Error finding container 56e90c8efbe401e94636f6afd5b4c7bf747d5a2a6491f0c586b68699a8b7393a: Status 404 returned error can't find the container with id 56e90c8efbe401e94636f6afd5b4c7bf747d5a2a6491f0c586b68699a8b7393a Aug 01 00:39:35.448500 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:35.448449 2568 generic.go:356] "Generic (PLEG): container finished" podID="cde15bfa-85a5-4c51-b671-a52e95534f8f" containerID="2f4fbca71e5db998aa68b48600672ff1d9343a0eebdd399327075cad17f99114" exitCode=0 Aug 01 00:39:35.448500 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:35.448492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cde15bfa-85a5-4c51-b671-a52e95534f8f","Type":"ContainerDied","Data":"2f4fbca71e5db998aa68b48600672ff1d9343a0eebdd399327075cad17f99114"} Aug 01 00:39:35.448745 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:35.448540 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cde15bfa-85a5-4c51-b671-a52e95534f8f","Type":"ContainerStarted","Data":"56e90c8efbe401e94636f6afd5b4c7bf747d5a2a6491f0c586b68699a8b7393a"} Aug 01 00:39:36.229312 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.229283 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d742d153-2e24-4cb2-9422-275facf5e706" path="/var/lib/kubelet/pods/d742d153-2e24-4cb2-9422-275facf5e706/volumes" Aug 01 00:39:36.458818 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.458782 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cde15bfa-85a5-4c51-b671-a52e95534f8f","Type":"ContainerStarted","Data":"b657f55f6bd3aa19d6e372e7b46f24a8cec9fcc53cd67dd9bc7eeb51dff64e1f"} Aug 01 00:39:36.458975 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.458854 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cde15bfa-85a5-4c51-b671-a52e95534f8f","Type":"ContainerStarted","Data":"a83486230f626edc0922f51ab7a20c2b14834c02afa9f8ab9a9cfeb5132c5d91"} Aug 01 00:39:36.458975 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.458882 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cde15bfa-85a5-4c51-b671-a52e95534f8f","Type":"ContainerStarted","Data":"8a5f6daa4146627ed554adcf348716966511d1903bd8a3db38a35660991eca9d"} Aug 01 00:39:36.458975 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.458905 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cde15bfa-85a5-4c51-b671-a52e95534f8f","Type":"ContainerStarted","Data":"8b1eca3af56eef861c4a6cece34f4207db949c97eb50737cb5352988f9da5be5"} Aug 01 00:39:36.458975 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.458937 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cde15bfa-85a5-4c51-b671-a52e95534f8f","Type":"ContainerStarted","Data":"3899f69f67e130904bf917898f8dbfe30f77012a57b382474c16635ab07324d5"} Aug 01 00:39:36.458975 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.458961 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cde15bfa-85a5-4c51-b671-a52e95534f8f","Type":"ContainerStarted","Data":"17445d0200d8b008319b5833a0d4a59d7dff0295faf35549eb1e936f3d9bd832"} Aug 01 00:39:36.485408 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.485324 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.469948455 podStartE2EDuration="2.485309269s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.360814ms" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:36.483402598 +0000 UTC m=+354.693031076" watchObservedRunningTime="2026-08-01 00:39:36.485309269 +0000 UTC m=+354.694937745" Aug 01 00:39:36.488332 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.488306 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d"] Aug 01 00:39:36.491429 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.491414 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.493507 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.493488 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 01 00:39:36.493610 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.493544 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 01 00:39:36.494125 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.493964 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 01 00:39:36.494125 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.493986 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 01 00:39:36.494125 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.494011 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z6zpz\"" Aug 01 00:39:36.495000 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.494980 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 01 00:39:36.499048 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.499027 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 01 00:39:36.504879 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.504851 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d"] Aug 01 00:39:36.586630 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.586597 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.586801 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.586781 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-metrics-client-ca\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.586928 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.586912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4t62\" (UniqueName: \"kubernetes.io/projected/26b55d53-5bdf-454e-939e-50521ca86047-kube-api-access-b4t62\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.587123 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.587092 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-telemeter-client-tls\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.587200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.587144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-serving-certs-ca-bundle\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.587200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.587186 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.587352 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.587254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-secret-telemeter-client\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.587352 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.587272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-federate-client-tls\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.688025 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.687985 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.688025 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.688028 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-secret-telemeter-client\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.688290 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.688047 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-federate-client-tls\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.688290 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.688170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.688290 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.688238 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-metrics-client-ca\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.688430 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.688307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4t62\" (UniqueName: \"kubernetes.io/projected/26b55d53-5bdf-454e-939e-50521ca86047-kube-api-access-b4t62\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.688430 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.688354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-telemeter-client-tls\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.689147 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.688790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-serving-certs-ca-bundle\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.689461 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.689433 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-metrics-client-ca\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.689558 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.689437 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-serving-certs-ca-bundle\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.689631 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.689553 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26b55d53-5bdf-454e-939e-50521ca86047-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.692388 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.692366 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-federate-client-tls\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.693055 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.693027 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.693329 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.693311 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-telemeter-client-tls\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.696851 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.696710 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/26b55d53-5bdf-454e-939e-50521ca86047-secret-telemeter-client\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.697159 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.697142 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4t62\" (UniqueName: \"kubernetes.io/projected/26b55d53-5bdf-454e-939e-50521ca86047-kube-api-access-b4t62\") pod \"telemeter-client-bdd84f7c5-2wn9d\" (UID: \"26b55d53-5bdf-454e-939e-50521ca86047\") " pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.802640 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.802559 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" Aug 01 00:39:36.933993 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.933966 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d"] Aug 01 00:39:36.936250 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:39:36.936205 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b55d53_5bdf_454e_939e_50521ca86047.slice/crio-db2458dc9497bc27a98438db20f7bed8c8cdcf5d37be78f4e5702be2aedd30b6 WatchSource:0}: Error finding container db2458dc9497bc27a98438db20f7bed8c8cdcf5d37be78f4e5702be2aedd30b6: Status 404 returned error can't find the container with id db2458dc9497bc27a98438db20f7bed8c8cdcf5d37be78f4e5702be2aedd30b6 Aug 01 00:39:36.938116 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:36.938101 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:39:37.464836 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:37.464800 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" event={"ID":"26b55d53-5bdf-454e-939e-50521ca86047","Type":"ContainerStarted","Data":"db2458dc9497bc27a98438db20f7bed8c8cdcf5d37be78f4e5702be2aedd30b6"} Aug 01 00:39:39.476137 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:39.476104 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" event={"ID":"26b55d53-5bdf-454e-939e-50521ca86047","Type":"ContainerStarted","Data":"eb9011d27e11b4dfdf2f96a1165d90414c426eb9738ede3b79a8b290bc773f46"} Aug 01 00:39:39.476514 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:39.476168 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" event={"ID":"26b55d53-5bdf-454e-939e-50521ca86047","Type":"ContainerStarted","Data":"0d00c77d602b72239aec784797cd38480f2e834c3cf59118847e9232dd35f662"} Aug 01 00:39:39.476514 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:39.476188 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" event={"ID":"26b55d53-5bdf-454e-939e-50521ca86047","Type":"ContainerStarted","Data":"281295b1d10ac440d280da3609ce5946bac01b05ab7652fdd6679f71d1c9dec7"} Aug 01 00:39:39.496776 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:39.496734 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-bdd84f7c5-2wn9d" podStartSLOduration=1.465518632 podStartE2EDuration="3.496718888s" totalImagesPullingTime="2.031200256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:39.495333316 +0000 UTC m=+357.704961771" watchObservedRunningTime="2026-08-01 00:39:39.496718888 +0000 UTC m=+357.706347348" Aug 01 00:39:55.147260 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:55.147202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:39:55.149583 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:55.149566 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf38574-98e0-464c-8f21-bdb577e4b0ec-metrics-certs\") pod \"network-metrics-daemon-249m9\" (UID: \"dcf38574-98e0-464c-8f21-bdb577e4b0ec\") " pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:39:55.427601 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:55.427505 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hdfd9\"" Aug 01 00:39:55.436367 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:55.436345 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-249m9" Aug 01 00:39:55.568292 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:55.568262 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-249m9"] Aug 01 00:39:55.571252 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:39:55.571205 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf38574_98e0_464c_8f21_bdb577e4b0ec.slice/crio-d88607bbbc8159face1725434eab2f202eb47669180186d596b0b8bc41a69c6e WatchSource:0}: Error finding container d88607bbbc8159face1725434eab2f202eb47669180186d596b0b8bc41a69c6e: Status 404 returned error can't find the container with id d88607bbbc8159face1725434eab2f202eb47669180186d596b0b8bc41a69c6e Aug 01 00:39:56.553489 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:56.553449 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-249m9" event={"ID":"dcf38574-98e0-464c-8f21-bdb577e4b0ec","Type":"ContainerStarted","Data":"d88607bbbc8159face1725434eab2f202eb47669180186d596b0b8bc41a69c6e"} Aug 01 00:39:57.559570 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:57.559532 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-249m9" event={"ID":"dcf38574-98e0-464c-8f21-bdb577e4b0ec","Type":"ContainerStarted","Data":"daa8430ba1062790a0a2c1361a5fadca282251508856a72df4421895f92f7e78"} Aug 01 00:39:57.559948 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:57.559595 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-249m9" event={"ID":"dcf38574-98e0-464c-8f21-bdb577e4b0ec","Type":"ContainerStarted","Data":"828b108cdb1079ff3447c3c2720807b9c0e67cd86ef5515f3763cb8d2b04c645"} Aug 01 00:39:57.577206 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:39:57.577139 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-249m9" podStartSLOduration=374.489554244 podStartE2EDuration="6m15.57712434s" totalImagesPullingTime="1.087570096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:57.575236563 +0000 UTC m=+375.784865020" watchObservedRunningTime="2026-08-01 00:39:57.57712434 +0000 UTC m=+375.786752846" Aug 01 00:41:41.828600 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.828558 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k"] Aug 01 00:41:41.832346 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.832325 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:41.834452 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.834432 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 01 00:41:41.835077 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.835060 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 01 00:41:41.835149 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.835062 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4pnxs\"" Aug 01 00:41:41.839735 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.839703 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k"] Aug 01 00:41:41.913834 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.913798 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:41.914017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.913847 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcwh\" (UniqueName: \"kubernetes.io/projected/6656f516-0fe9-4b09-9b0e-f87614712f59-kube-api-access-dgcwh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:41.914017 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:41.913894 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:42.014814 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.014779 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:42.014982 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.014831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcwh\" (UniqueName: \"kubernetes.io/projected/6656f516-0fe9-4b09-9b0e-f87614712f59-kube-api-access-dgcwh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:42.014982 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.014862 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:42.015246 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.015203 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:42.015303 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.015202 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:42.023332 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.023309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcwh\" (UniqueName: \"kubernetes.io/projected/6656f516-0fe9-4b09-9b0e-f87614712f59-kube-api-access-dgcwh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:42.144075 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.144000 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4pnxs\"" Aug 01 00:41:42.152667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.152644 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:41:42.286069 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.286041 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k"] Aug 01 00:41:42.289192 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:41:42.289158 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6656f516_0fe9_4b09_9b0e_f87614712f59.slice/crio-126c308e1e3b6852c9c89fbd4ad25d16b2adb3c2987aa46eae669b6f145360a9 WatchSource:0}: Error finding container 126c308e1e3b6852c9c89fbd4ad25d16b2adb3c2987aa46eae669b6f145360a9: Status 404 returned error can't find the container with id 126c308e1e3b6852c9c89fbd4ad25d16b2adb3c2987aa46eae669b6f145360a9 Aug 01 00:41:42.972984 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:42.972941 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" event={"ID":"6656f516-0fe9-4b09-9b0e-f87614712f59","Type":"ContainerStarted","Data":"126c308e1e3b6852c9c89fbd4ad25d16b2adb3c2987aa46eae669b6f145360a9"} Aug 01 00:41:49.009256 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:49.009150 2568 generic.go:356] "Generic (PLEG): container finished" podID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerID="52ca1b7d8fdc47a06931f0ccf6c1fca8e83d442012ba4b9015a8f4fbbb7e355a" exitCode=0 Aug 01 00:41:49.009256 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:49.009240 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" event={"ID":"6656f516-0fe9-4b09-9b0e-f87614712f59","Type":"ContainerDied","Data":"52ca1b7d8fdc47a06931f0ccf6c1fca8e83d442012ba4b9015a8f4fbbb7e355a"} Aug 01 00:41:52.023514 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:52.023483 2568 generic.go:356] "Generic (PLEG): container finished" podID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerID="007c2d3333cd238c9e0f29e918326a927259e9dd3034deb7c8c16c76f1f251a2" exitCode=0 Aug 01 00:41:52.023878 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:52.023572 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" event={"ID":"6656f516-0fe9-4b09-9b0e-f87614712f59","Type":"ContainerDied","Data":"007c2d3333cd238c9e0f29e918326a927259e9dd3034deb7c8c16c76f1f251a2"} Aug 01 00:41:59.057319 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:59.057283 2568 generic.go:356] "Generic (PLEG): container finished" podID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerID="dbf497676ec916be94875998c4491db697952879f5f247117639047dbcad327e" exitCode=0 Aug 01 00:41:59.057689 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:41:59.057346 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" event={"ID":"6656f516-0fe9-4b09-9b0e-f87614712f59","Type":"ContainerDied","Data":"dbf497676ec916be94875998c4491db697952879f5f247117639047dbcad327e"} Aug 01 00:42:00.186012 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.185986 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:42:00.274623 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.274592 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgcwh\" (UniqueName: \"kubernetes.io/projected/6656f516-0fe9-4b09-9b0e-f87614712f59-kube-api-access-dgcwh\") pod \"6656f516-0fe9-4b09-9b0e-f87614712f59\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " Aug 01 00:42:00.274782 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.274642 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-bundle\") pod \"6656f516-0fe9-4b09-9b0e-f87614712f59\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " Aug 01 00:42:00.274782 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.274739 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-util\") pod \"6656f516-0fe9-4b09-9b0e-f87614712f59\" (UID: \"6656f516-0fe9-4b09-9b0e-f87614712f59\") " Aug 01 00:42:00.275393 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.275327 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-bundle" (OuterVolumeSpecName: "bundle") pod "6656f516-0fe9-4b09-9b0e-f87614712f59" (UID: "6656f516-0fe9-4b09-9b0e-f87614712f59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:42:00.276927 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.276903 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6656f516-0fe9-4b09-9b0e-f87614712f59-kube-api-access-dgcwh" (OuterVolumeSpecName: "kube-api-access-dgcwh") pod "6656f516-0fe9-4b09-9b0e-f87614712f59" (UID: "6656f516-0fe9-4b09-9b0e-f87614712f59"). InnerVolumeSpecName "kube-api-access-dgcwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:42:00.279609 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.279589 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-util" (OuterVolumeSpecName: "util") pod "6656f516-0fe9-4b09-9b0e-f87614712f59" (UID: "6656f516-0fe9-4b09-9b0e-f87614712f59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:42:00.376029 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.375943 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcwh\" (UniqueName: \"kubernetes.io/projected/6656f516-0fe9-4b09-9b0e-f87614712f59-kube-api-access-dgcwh\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:00.376029 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.375967 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-bundle\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:00.376029 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:00.375979 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6656f516-0fe9-4b09-9b0e-f87614712f59-util\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:01.067315 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:01.067283 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" Aug 01 00:42:01.067315 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:01.067301 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qhr4k" event={"ID":"6656f516-0fe9-4b09-9b0e-f87614712f59","Type":"ContainerDied","Data":"126c308e1e3b6852c9c89fbd4ad25d16b2adb3c2987aa46eae669b6f145360a9"} Aug 01 00:42:01.067553 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:01.067343 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="126c308e1e3b6852c9c89fbd4ad25d16b2adb3c2987aa46eae669b6f145360a9" Aug 01 00:42:18.018071 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.018037 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5"] Aug 01 00:42:18.018567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.018416 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerName="util" Aug 01 00:42:18.018567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.018429 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerName="util" Aug 01 00:42:18.018567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.018437 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerName="extract" Aug 01 00:42:18.018567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.018444 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerName="extract" Aug 01 00:42:18.018567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.018455 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerName="pull" Aug 01 00:42:18.018567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.018461 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerName="pull" Aug 01 00:42:18.018567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.018527 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6656f516-0fe9-4b09-9b0e-f87614712f59" containerName="extract" Aug 01 00:42:18.021703 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.021687 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.023693 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.023669 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 01 00:42:18.023693 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.023683 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4pnxs\"" Aug 01 00:42:18.023865 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.023717 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 01 00:42:18.027901 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.027879 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5"] Aug 01 00:42:18.114594 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.114545 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.114594 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.114588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmvx\" (UniqueName: \"kubernetes.io/projected/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-kube-api-access-6tmvx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.114807 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.114608 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.215502 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.215464 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmvx\" (UniqueName: \"kubernetes.io/projected/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-kube-api-access-6tmvx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.215502 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.215508 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.215742 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.215583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.215909 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.215890 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.215959 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.215929 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.233472 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.233433 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmvx\" (UniqueName: \"kubernetes.io/projected/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-kube-api-access-6tmvx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.331512 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.331429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:18.462150 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.462125 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5"] Aug 01 00:42:18.464675 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:42:18.464648 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281208bd_2e8d_4d5c_9837_2c4c5d0cb416.slice/crio-362dd531e1aaefc7fb532c97e9034e31f4d5dd05bd12ab234eb498d365c10149 WatchSource:0}: Error finding container 362dd531e1aaefc7fb532c97e9034e31f4d5dd05bd12ab234eb498d365c10149: Status 404 returned error can't find the container with id 362dd531e1aaefc7fb532c97e9034e31f4d5dd05bd12ab234eb498d365c10149 Aug 01 00:42:18.651921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.651837 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h"] Aug 01 00:42:18.655106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.655084 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" Aug 01 00:42:18.657681 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.657662 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vvjq8\"" Aug 01 00:42:18.658299 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.658283 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 01 00:42:18.658350 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.658297 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 01 00:42:18.663717 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.663693 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h"] Aug 01 00:42:18.720363 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.720327 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2a09277-60c5-4943-8738-eb28a9d8a48f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j7m7h\" (UID: \"c2a09277-60c5-4943-8738-eb28a9d8a48f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" Aug 01 00:42:18.720552 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.720382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj72d\" (UniqueName: \"kubernetes.io/projected/c2a09277-60c5-4943-8738-eb28a9d8a48f-kube-api-access-sj72d\") pod \"cert-manager-cainjector-559cdd8c87-j7m7h\" (UID: \"c2a09277-60c5-4943-8738-eb28a9d8a48f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" Aug 01 00:42:18.821620 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.821574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2a09277-60c5-4943-8738-eb28a9d8a48f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j7m7h\" (UID: \"c2a09277-60c5-4943-8738-eb28a9d8a48f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" Aug 01 00:42:18.821781 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.821642 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj72d\" (UniqueName: \"kubernetes.io/projected/c2a09277-60c5-4943-8738-eb28a9d8a48f-kube-api-access-sj72d\") pod \"cert-manager-cainjector-559cdd8c87-j7m7h\" (UID: \"c2a09277-60c5-4943-8738-eb28a9d8a48f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" Aug 01 00:42:18.829973 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.829957 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2a09277-60c5-4943-8738-eb28a9d8a48f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j7m7h\" (UID: \"c2a09277-60c5-4943-8738-eb28a9d8a48f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" Aug 01 00:42:18.830080 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.830010 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj72d\" (UniqueName: \"kubernetes.io/projected/c2a09277-60c5-4943-8738-eb28a9d8a48f-kube-api-access-sj72d\") pod \"cert-manager-cainjector-559cdd8c87-j7m7h\" (UID: \"c2a09277-60c5-4943-8738-eb28a9d8a48f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" Aug 01 00:42:18.985054 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:18.985022 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" Aug 01 00:42:19.115360 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:19.115322 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h"] Aug 01 00:42:19.118163 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:42:19.118130 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2a09277_60c5_4943_8738_eb28a9d8a48f.slice/crio-252605804a39f1518e92204e5f1c4cc4cf331dc18118458e78b33b4ac0b4ce05 WatchSource:0}: Error finding container 252605804a39f1518e92204e5f1c4cc4cf331dc18118458e78b33b4ac0b4ce05: Status 404 returned error can't find the container with id 252605804a39f1518e92204e5f1c4cc4cf331dc18118458e78b33b4ac0b4ce05 Aug 01 00:42:19.139946 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:19.139916 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" event={"ID":"c2a09277-60c5-4943-8738-eb28a9d8a48f","Type":"ContainerStarted","Data":"252605804a39f1518e92204e5f1c4cc4cf331dc18118458e78b33b4ac0b4ce05"} Aug 01 00:42:19.141425 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:19.141402 2568 generic.go:356] "Generic (PLEG): container finished" podID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerID="46266f2b12695be92bc267069cb851c7d431471671f9aa491a857ff7b5f70fd2" exitCode=0 Aug 01 00:42:19.141527 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:19.141487 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" event={"ID":"281208bd-2e8d-4d5c-9837-2c4c5d0cb416","Type":"ContainerDied","Data":"46266f2b12695be92bc267069cb851c7d431471671f9aa491a857ff7b5f70fd2"} Aug 01 00:42:19.141573 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:19.141529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" event={"ID":"281208bd-2e8d-4d5c-9837-2c4c5d0cb416","Type":"ContainerStarted","Data":"362dd531e1aaefc7fb532c97e9034e31f4d5dd05bd12ab234eb498d365c10149"} Aug 01 00:42:22.158675 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:22.158634 2568 generic.go:356] "Generic (PLEG): container finished" podID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerID="cc0e6ab7a06fbb6da05b089dd37f7197ad95cf44451f329104da5beb8aae7c27" exitCode=0 Aug 01 00:42:22.159175 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:22.158726 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" event={"ID":"281208bd-2e8d-4d5c-9837-2c4c5d0cb416","Type":"ContainerDied","Data":"cc0e6ab7a06fbb6da05b089dd37f7197ad95cf44451f329104da5beb8aae7c27"} Aug 01 00:42:23.164491 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:23.164444 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" event={"ID":"c2a09277-60c5-4943-8738-eb28a9d8a48f","Type":"ContainerStarted","Data":"dd5fce7024dd759d29c29b20f19649b25ade280d1dc30b24a349c80310ec626d"} Aug 01 00:42:23.166579 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:23.166554 2568 generic.go:356] "Generic (PLEG): container finished" podID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerID="3c12b5aff59dd290a0cf819780ed09b018205336c34df0960e4ffa33780e6678" exitCode=0 Aug 01 00:42:23.166686 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:23.166636 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" event={"ID":"281208bd-2e8d-4d5c-9837-2c4c5d0cb416","Type":"ContainerDied","Data":"3c12b5aff59dd290a0cf819780ed09b018205336c34df0960e4ffa33780e6678"} Aug 01 00:42:23.182959 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:23.182907 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j7m7h" podStartSLOduration=1.8352613199999999 podStartE2EDuration="5.182889609s" totalImagesPullingTime="3.347628289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:23.180449842 +0000 UTC m=+521.390078299" watchObservedRunningTime="2026-08-01 00:42:23.182889609 +0000 UTC m=+521.392518066" Aug 01 00:42:24.303747 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.303720 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:24.372701 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.372665 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-util\") pod \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " Aug 01 00:42:24.372866 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.372730 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-bundle\") pod \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " Aug 01 00:42:24.372866 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.372750 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tmvx\" (UniqueName: \"kubernetes.io/projected/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-kube-api-access-6tmvx\") pod \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\" (UID: \"281208bd-2e8d-4d5c-9837-2c4c5d0cb416\") " Aug 01 00:42:24.373123 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.373100 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-bundle" (OuterVolumeSpecName: "bundle") pod "281208bd-2e8d-4d5c-9837-2c4c5d0cb416" (UID: "281208bd-2e8d-4d5c-9837-2c4c5d0cb416"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:42:24.375010 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.374984 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-kube-api-access-6tmvx" (OuterVolumeSpecName: "kube-api-access-6tmvx") pod "281208bd-2e8d-4d5c-9837-2c4c5d0cb416" (UID: "281208bd-2e8d-4d5c-9837-2c4c5d0cb416"). InnerVolumeSpecName "kube-api-access-6tmvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:42:24.377461 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.377439 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-util" (OuterVolumeSpecName: "util") pod "281208bd-2e8d-4d5c-9837-2c4c5d0cb416" (UID: "281208bd-2e8d-4d5c-9837-2c4c5d0cb416"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:42:24.473729 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.473690 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-util\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:24.473729 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.473723 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-bundle\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:24.473729 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:24.473736 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmvx\" (UniqueName: \"kubernetes.io/projected/281208bd-2e8d-4d5c-9837-2c4c5d0cb416-kube-api-access-6tmvx\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:25.177532 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:25.177499 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" Aug 01 00:42:25.177726 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:25.177504 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frltf5" event={"ID":"281208bd-2e8d-4d5c-9837-2c4c5d0cb416","Type":"ContainerDied","Data":"362dd531e1aaefc7fb532c97e9034e31f4d5dd05bd12ab234eb498d365c10149"} Aug 01 00:42:25.177726 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:25.177617 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="362dd531e1aaefc7fb532c97e9034e31f4d5dd05bd12ab234eb498d365c10149" Aug 01 00:42:31.129720 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.129680 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l"] Aug 01 00:42:31.130143 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.130006 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerName="util" Aug 01 00:42:31.130143 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.130016 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerName="util" Aug 01 00:42:31.130143 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.130025 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerName="extract" Aug 01 00:42:31.130143 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.130030 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerName="extract" Aug 01 00:42:31.130143 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.130040 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerName="pull" Aug 01 00:42:31.130143 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.130045 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerName="pull" Aug 01 00:42:31.130143 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.130101 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="281208bd-2e8d-4d5c-9837-2c4c5d0cb416" containerName="extract" Aug 01 00:42:31.133972 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.133956 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" Aug 01 00:42:31.136128 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.136097 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-484vj\"" Aug 01 00:42:31.136257 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.136107 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 01 00:42:31.136257 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.136117 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:42:31.143249 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.143207 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l"] Aug 01 00:42:31.230614 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.230573 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c1001d-cf73-4cfd-84a0-e3159d18e674-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jb79l\" (UID: \"97c1001d-cf73-4cfd-84a0-e3159d18e674\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" Aug 01 00:42:31.230813 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.230614 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vm5w\" (UniqueName: \"kubernetes.io/projected/97c1001d-cf73-4cfd-84a0-e3159d18e674-kube-api-access-2vm5w\") pod \"openshift-lws-operator-fd8ccff4c-jb79l\" (UID: \"97c1001d-cf73-4cfd-84a0-e3159d18e674\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" Aug 01 00:42:31.331401 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.331363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c1001d-cf73-4cfd-84a0-e3159d18e674-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jb79l\" (UID: \"97c1001d-cf73-4cfd-84a0-e3159d18e674\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" Aug 01 00:42:31.331401 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.331400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vm5w\" (UniqueName: \"kubernetes.io/projected/97c1001d-cf73-4cfd-84a0-e3159d18e674-kube-api-access-2vm5w\") pod \"openshift-lws-operator-fd8ccff4c-jb79l\" (UID: \"97c1001d-cf73-4cfd-84a0-e3159d18e674\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" Aug 01 00:42:31.331754 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.331731 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97c1001d-cf73-4cfd-84a0-e3159d18e674-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jb79l\" (UID: \"97c1001d-cf73-4cfd-84a0-e3159d18e674\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" Aug 01 00:42:31.339291 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.339267 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vm5w\" (UniqueName: \"kubernetes.io/projected/97c1001d-cf73-4cfd-84a0-e3159d18e674-kube-api-access-2vm5w\") pod \"openshift-lws-operator-fd8ccff4c-jb79l\" (UID: \"97c1001d-cf73-4cfd-84a0-e3159d18e674\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" Aug 01 00:42:31.443363 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.443330 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" Aug 01 00:42:31.582509 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:31.582479 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l"] Aug 01 00:42:31.584605 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:42:31.584575 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97c1001d_cf73_4cfd_84a0_e3159d18e674.slice/crio-ee27b5124987363c3fa0a0893590aa1f7efd8decc403bd81e468b21b219de592 WatchSource:0}: Error finding container ee27b5124987363c3fa0a0893590aa1f7efd8decc403bd81e468b21b219de592: Status 404 returned error can't find the container with id ee27b5124987363c3fa0a0893590aa1f7efd8decc403bd81e468b21b219de592 Aug 01 00:42:32.210170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:32.210133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" event={"ID":"97c1001d-cf73-4cfd-84a0-e3159d18e674","Type":"ContainerStarted","Data":"ee27b5124987363c3fa0a0893590aa1f7efd8decc403bd81e468b21b219de592"} Aug 01 00:42:34.222242 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:34.222183 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" event={"ID":"97c1001d-cf73-4cfd-84a0-e3159d18e674","Type":"ContainerStarted","Data":"f7b570da6ceae4e2761670aa0807b7ddc123fc22081369dc838cfbd1517786f8"} Aug 01 00:42:34.237096 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:34.237036 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jb79l" podStartSLOduration=0.673939663 podStartE2EDuration="3.237019775s" totalImagesPullingTime="2.563080112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:42:34.236276971 +0000 UTC m=+532.445905428" watchObservedRunningTime="2026-08-01 00:42:34.237019775 +0000 UTC m=+532.446648232" Aug 01 00:42:36.914448 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.914411 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s"] Aug 01 00:42:36.917966 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.917951 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:36.920091 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.920070 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 01 00:42:36.920185 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.920072 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 01 00:42:36.920185 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.920074 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4pnxs\"" Aug 01 00:42:36.925492 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.925472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s"] Aug 01 00:42:36.984913 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.984878 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cntk\" (UniqueName: \"kubernetes.io/projected/7450b54f-da23-4752-8323-18f2139df244-kube-api-access-6cntk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:36.985094 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.984928 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:36.985094 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:36.985003 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:37.085590 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:37.085550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:37.085789 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:37.085609 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:37.085789 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:37.085663 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cntk\" (UniqueName: \"kubernetes.io/projected/7450b54f-da23-4752-8323-18f2139df244-kube-api-access-6cntk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:37.085997 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:37.085971 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:37.086057 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:37.085992 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:37.095184 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:37.095159 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cntk\" (UniqueName: \"kubernetes.io/projected/7450b54f-da23-4752-8323-18f2139df244-kube-api-access-6cntk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:37.227468 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:37.227432 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:37.362512 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:37.362468 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s"] Aug 01 00:42:37.367536 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:42:37.367508 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7450b54f_da23_4752_8323_18f2139df244.slice/crio-d4a548d83c01ab7c6b79901b6024d32fdc30bc9953042e12b3c74b40b02fc3d8 WatchSource:0}: Error finding container d4a548d83c01ab7c6b79901b6024d32fdc30bc9953042e12b3c74b40b02fc3d8: Status 404 returned error can't find the container with id d4a548d83c01ab7c6b79901b6024d32fdc30bc9953042e12b3c74b40b02fc3d8 Aug 01 00:42:38.243765 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:38.243729 2568 generic.go:356] "Generic (PLEG): container finished" podID="7450b54f-da23-4752-8323-18f2139df244" containerID="0322280eb098a97c9dc9f7552b3d09d7d04b114349fa1b2e095c7fce75ddc9dc" exitCode=0 Aug 01 00:42:38.244127 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:38.243812 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" event={"ID":"7450b54f-da23-4752-8323-18f2139df244","Type":"ContainerDied","Data":"0322280eb098a97c9dc9f7552b3d09d7d04b114349fa1b2e095c7fce75ddc9dc"} Aug 01 00:42:38.244127 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:38.243857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" event={"ID":"7450b54f-da23-4752-8323-18f2139df244","Type":"ContainerStarted","Data":"d4a548d83c01ab7c6b79901b6024d32fdc30bc9953042e12b3c74b40b02fc3d8"} Aug 01 00:42:39.250941 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:39.250908 2568 generic.go:356] "Generic (PLEG): container finished" podID="7450b54f-da23-4752-8323-18f2139df244" containerID="6ec03988423f7fb263f5031e457045eae9dacfae40b008a62e10ef235c16e82d" exitCode=0 Aug 01 00:42:39.251439 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:39.250965 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" event={"ID":"7450b54f-da23-4752-8323-18f2139df244","Type":"ContainerDied","Data":"6ec03988423f7fb263f5031e457045eae9dacfae40b008a62e10ef235c16e82d"} Aug 01 00:42:40.257753 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:40.257708 2568 generic.go:356] "Generic (PLEG): container finished" podID="7450b54f-da23-4752-8323-18f2139df244" containerID="571361681a2518ec2330659fecb3721f58787b4b9ab7dd5fd110f377d0ea940a" exitCode=0 Aug 01 00:42:40.258191 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:40.257775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" event={"ID":"7450b54f-da23-4752-8323-18f2139df244","Type":"ContainerDied","Data":"571361681a2518ec2330659fecb3721f58787b4b9ab7dd5fd110f377d0ea940a"} Aug 01 00:42:41.386246 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.386195 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:41.519287 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.519158 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-bundle\") pod \"7450b54f-da23-4752-8323-18f2139df244\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " Aug 01 00:42:41.519287 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.519241 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-util\") pod \"7450b54f-da23-4752-8323-18f2139df244\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " Aug 01 00:42:41.519287 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.519276 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cntk\" (UniqueName: \"kubernetes.io/projected/7450b54f-da23-4752-8323-18f2139df244-kube-api-access-6cntk\") pod \"7450b54f-da23-4752-8323-18f2139df244\" (UID: \"7450b54f-da23-4752-8323-18f2139df244\") " Aug 01 00:42:41.519896 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.519871 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-bundle" (OuterVolumeSpecName: "bundle") pod "7450b54f-da23-4752-8323-18f2139df244" (UID: "7450b54f-da23-4752-8323-18f2139df244"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:42:41.521725 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.521703 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7450b54f-da23-4752-8323-18f2139df244-kube-api-access-6cntk" (OuterVolumeSpecName: "kube-api-access-6cntk") pod "7450b54f-da23-4752-8323-18f2139df244" (UID: "7450b54f-da23-4752-8323-18f2139df244"). InnerVolumeSpecName "kube-api-access-6cntk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:42:41.525473 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.525448 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-util" (OuterVolumeSpecName: "util") pod "7450b54f-da23-4752-8323-18f2139df244" (UID: "7450b54f-da23-4752-8323-18f2139df244"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:42:41.619878 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.619845 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-bundle\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:41.619878 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.619875 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7450b54f-da23-4752-8323-18f2139df244-util\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:41.619878 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:41.619884 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cntk\" (UniqueName: \"kubernetes.io/projected/7450b54f-da23-4752-8323-18f2139df244-kube-api-access-6cntk\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:42:42.269171 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:42.269138 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" event={"ID":"7450b54f-da23-4752-8323-18f2139df244","Type":"ContainerDied","Data":"d4a548d83c01ab7c6b79901b6024d32fdc30bc9953042e12b3c74b40b02fc3d8"} Aug 01 00:42:42.269171 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:42.269159 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afh845s" Aug 01 00:42:42.269393 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:42.269183 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4a548d83c01ab7c6b79901b6024d32fdc30bc9953042e12b3c74b40b02fc3d8" Aug 01 00:42:53.600910 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.600876 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg"] Aug 01 00:42:53.601404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.601203 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7450b54f-da23-4752-8323-18f2139df244" containerName="pull" Aug 01 00:42:53.601404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.601227 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7450b54f-da23-4752-8323-18f2139df244" containerName="pull" Aug 01 00:42:53.601404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.601244 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7450b54f-da23-4752-8323-18f2139df244" containerName="util" Aug 01 00:42:53.601404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.601249 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7450b54f-da23-4752-8323-18f2139df244" containerName="util" Aug 01 00:42:53.601404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.601259 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7450b54f-da23-4752-8323-18f2139df244" containerName="extract" Aug 01 00:42:53.601404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.601265 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7450b54f-da23-4752-8323-18f2139df244" containerName="extract" Aug 01 00:42:53.601404 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.601335 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7450b54f-da23-4752-8323-18f2139df244" containerName="extract" Aug 01 00:42:53.607667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.607649 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.610864 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.610841 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 01 00:42:53.610982 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.610873 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 01 00:42:53.610982 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.610838 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 01 00:42:53.611076 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.611036 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tj7sc\"" Aug 01 00:42:53.611353 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.611341 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 01 00:42:53.641105 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.641078 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg"] Aug 01 00:42:53.718059 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.718026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/241cfc9b-d570-474b-8f0e-1b0a38036189-manifests\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.718243 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.718074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/241cfc9b-d570-474b-8f0e-1b0a38036189-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.718243 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.718105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtmtv\" (UniqueName: \"kubernetes.io/projected/241cfc9b-d570-474b-8f0e-1b0a38036189-kube-api-access-mtmtv\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.718243 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.718146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/241cfc9b-d570-474b-8f0e-1b0a38036189-tmp\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.718243 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.718169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/241cfc9b-d570-474b-8f0e-1b0a38036189-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.819353 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.819319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/241cfc9b-d570-474b-8f0e-1b0a38036189-tmp\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.819353 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.819353 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/241cfc9b-d570-474b-8f0e-1b0a38036189-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.819551 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.819399 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/241cfc9b-d570-474b-8f0e-1b0a38036189-manifests\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.819551 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.819445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/241cfc9b-d570-474b-8f0e-1b0a38036189-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.819551 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.819488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtmtv\" (UniqueName: \"kubernetes.io/projected/241cfc9b-d570-474b-8f0e-1b0a38036189-kube-api-access-mtmtv\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.819837 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.819811 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/241cfc9b-d570-474b-8f0e-1b0a38036189-manifests\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.821920 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.821888 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/241cfc9b-d570-474b-8f0e-1b0a38036189-tmp\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.822074 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.822054 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/241cfc9b-d570-474b-8f0e-1b0a38036189-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.822115 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.822054 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/241cfc9b-d570-474b-8f0e-1b0a38036189-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.826833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.826807 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtmtv\" (UniqueName: \"kubernetes.io/projected/241cfc9b-d570-474b-8f0e-1b0a38036189-kube-api-access-mtmtv\") pod \"opendatahub-operator-controller-manager-64f88dcc48-sj8bg\" (UID: \"241cfc9b-d570-474b-8f0e-1b0a38036189\") " pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:53.917936 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:53.917851 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:42:54.059980 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:54.059956 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg"] Aug 01 00:42:54.062202 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:42:54.062168 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241cfc9b_d570_474b_8f0e_1b0a38036189.slice/crio-786991f6b6c11161bbd2f1e4ff71bf73d8890290ac60235d23e3350610662a24 WatchSource:0}: Error finding container 786991f6b6c11161bbd2f1e4ff71bf73d8890290ac60235d23e3350610662a24: Status 404 returned error can't find the container with id 786991f6b6c11161bbd2f1e4ff71bf73d8890290ac60235d23e3350610662a24 Aug 01 00:42:54.323889 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:54.323857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" event={"ID":"241cfc9b-d570-474b-8f0e-1b0a38036189","Type":"ContainerStarted","Data":"786991f6b6c11161bbd2f1e4ff71bf73d8890290ac60235d23e3350610662a24"} Aug 01 00:42:58.345512 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:58.345476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" event={"ID":"241cfc9b-d570-474b-8f0e-1b0a38036189","Type":"ContainerStarted","Data":"7a50cb4e900950b66633e8e2489aecf7dc8b2407b4495ad669bb27869cb4b7ad"} Aug 01 00:42:59.351665 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:59.351631 2568 generic.go:356] "Generic (PLEG): container finished" podID="241cfc9b-d570-474b-8f0e-1b0a38036189" containerID="7a50cb4e900950b66633e8e2489aecf7dc8b2407b4495ad669bb27869cb4b7ad" exitCode=0 Aug 01 00:42:59.352086 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:42:59.351713 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" event={"ID":"241cfc9b-d570-474b-8f0e-1b0a38036189","Type":"ContainerDied","Data":"7a50cb4e900950b66633e8e2489aecf7dc8b2407b4495ad669bb27869cb4b7ad"} Aug 01 00:43:00.358516 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:00.358476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" event={"ID":"241cfc9b-d570-474b-8f0e-1b0a38036189","Type":"ContainerStarted","Data":"53788fff8c17138cc479df3564c422c64d8399740333bf3224efcc8f3398972a"} Aug 01 00:43:00.358911 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:00.358556 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:43:00.381870 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:00.381813 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" podStartSLOduration=2.598969787 podStartE2EDuration="7.381797786s" totalImagesPullingTime="3.70047178s" totalInitContainerRuntime="1.082356219s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:42:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:00.380143397 +0000 UTC m=+558.589771866" watchObservedRunningTime="2026-08-01 00:43:00.381797786 +0000 UTC m=+558.591426241" Aug 01 00:43:11.365996 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:11.365964 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64f88dcc48-sj8bg" Aug 01 00:43:13.637341 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.637305 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6vffs"] Aug 01 00:43:13.643061 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.643042 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:13.645272 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.645254 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-szr7k\"" Aug 01 00:43:13.645530 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.645511 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 01 00:43:13.651525 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.651502 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6vffs"] Aug 01 00:43:13.689161 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.689129 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ww2x\" (UniqueName: \"kubernetes.io/projected/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-kube-api-access-8ww2x\") pod \"odh-model-controller-7998c67dc4-6vffs\" (UID: \"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465\") " pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:13.689161 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.689166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-cert\") pod \"odh-model-controller-7998c67dc4-6vffs\" (UID: \"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465\") " pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:13.790312 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.790269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww2x\" (UniqueName: \"kubernetes.io/projected/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-kube-api-access-8ww2x\") pod \"odh-model-controller-7998c67dc4-6vffs\" (UID: \"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465\") " pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:13.790312 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.790316 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-cert\") pod \"odh-model-controller-7998c67dc4-6vffs\" (UID: \"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465\") " pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:13.790517 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:43:13.790440 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 01 00:43:13.790517 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:43:13.790512 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-cert podName:8a7adf51-7ecd-4cf0-b5b5-31fda87f0465 nodeName:}" failed. No retries permitted until 2026-08-01 00:43:14.290489956 +0000 UTC m=+572.500118390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-cert") pod "odh-model-controller-7998c67dc4-6vffs" (UID: "8a7adf51-7ecd-4cf0-b5b5-31fda87f0465") : secret "odh-model-controller-webhook-cert" not found Aug 01 00:43:13.806631 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:13.806594 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww2x\" (UniqueName: \"kubernetes.io/projected/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-kube-api-access-8ww2x\") pod \"odh-model-controller-7998c67dc4-6vffs\" (UID: \"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465\") " pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:14.295569 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:14.295535 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-cert\") pod \"odh-model-controller-7998c67dc4-6vffs\" (UID: \"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465\") " pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:14.298346 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:14.298314 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7adf51-7ecd-4cf0-b5b5-31fda87f0465-cert\") pod \"odh-model-controller-7998c67dc4-6vffs\" (UID: \"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465\") " pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:14.554665 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:14.554569 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:14.689028 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:14.688970 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6vffs"] Aug 01 00:43:14.691264 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:43:14.691236 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7adf51_7ecd_4cf0_b5b5_31fda87f0465.slice/crio-93a121257323725debda3be01d6687b95442154e02b42461c6c9ca5763b89048 WatchSource:0}: Error finding container 93a121257323725debda3be01d6687b95442154e02b42461c6c9ca5763b89048: Status 404 returned error can't find the container with id 93a121257323725debda3be01d6687b95442154e02b42461c6c9ca5763b89048 Aug 01 00:43:15.430153 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:15.430116 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" event={"ID":"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465","Type":"ContainerStarted","Data":"93a121257323725debda3be01d6687b95442154e02b42461c6c9ca5763b89048"} Aug 01 00:43:18.448384 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:18.448347 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" event={"ID":"8a7adf51-7ecd-4cf0-b5b5-31fda87f0465","Type":"ContainerStarted","Data":"ce537b724cae363a8a4eb1888e9adee4143108e7c038d84d99fc05a9a7626d32"} Aug 01 00:43:18.448746 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:18.448478 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:18.462854 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:18.462803 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" podStartSLOduration=2.49442241 podStartE2EDuration="5.462786166s" totalImagesPullingTime="2.968363756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:18.46254427 +0000 UTC m=+576.672172727" watchObservedRunningTime="2026-08-01 00:43:18.462786166 +0000 UTC m=+576.672414624" Aug 01 00:43:19.009812 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.009784 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph"] Aug 01 00:43:19.013275 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.013255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.015615 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.015594 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 01 00:43:19.015821 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.015802 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vzwfw\"" Aug 01 00:43:19.016078 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.015951 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 01 00:43:19.023864 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.023842 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph"] Aug 01 00:43:19.140435 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.140405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgl65\" (UniqueName: \"kubernetes.io/projected/b1c44826-c266-422d-94d6-658e0247af81-kube-api-access-xgl65\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.140613 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.140452 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c44826-c266-422d-94d6-658e0247af81-tls-certs\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.140613 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.140554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c44826-c266-422d-94d6-658e0247af81-tmp\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.241829 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.241796 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c44826-c266-422d-94d6-658e0247af81-tls-certs\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.241986 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.241852 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c44826-c266-422d-94d6-658e0247af81-tmp\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.241986 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.241888 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgl65\" (UniqueName: \"kubernetes.io/projected/b1c44826-c266-422d-94d6-658e0247af81-kube-api-access-xgl65\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.244448 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.244420 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c44826-c266-422d-94d6-658e0247af81-tmp\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.244587 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.244568 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c44826-c266-422d-94d6-658e0247af81-tls-certs\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.253298 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.253280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgl65\" (UniqueName: \"kubernetes.io/projected/b1c44826-c266-422d-94d6-658e0247af81-kube-api-access-xgl65\") pod \"kube-auth-proxy-7798dfdbd7-m5lph\" (UID: \"b1c44826-c266-422d-94d6-658e0247af81\") " pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.324320 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.324243 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" Aug 01 00:43:19.477040 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:19.477000 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph"] Aug 01 00:43:19.480092 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:43:19.480050 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c44826_c266_422d_94d6_658e0247af81.slice/crio-011e2b31197ad56219340dbf9c532e739c1dd3cc4a21b4800d1190932195c2fd WatchSource:0}: Error finding container 011e2b31197ad56219340dbf9c532e739c1dd3cc4a21b4800d1190932195c2fd: Status 404 returned error can't find the container with id 011e2b31197ad56219340dbf9c532e739c1dd3cc4a21b4800d1190932195c2fd Aug 01 00:43:20.101373 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.101333 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h"] Aug 01 00:43:20.105565 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.105542 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.107961 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.107930 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 01 00:43:20.107961 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.107944 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 01 00:43:20.108123 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.108106 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lkkdd\"" Aug 01 00:43:20.108359 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.108345 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 01 00:43:20.108864 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.108846 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 01 00:43:20.126560 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.126539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h"] Aug 01 00:43:20.148808 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.148770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c60356f-e050-4ddf-b82b-6298b88decca-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.148975 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.148827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.148975 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.148900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.149086 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.148980 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.149086 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.149057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.149182 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.149166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-kube-api-access-flcdx\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.149251 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.149201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.250040 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.249989 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c60356f-e050-4ddf-b82b-6298b88decca-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.250252 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.250085 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.250252 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.250152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.250252 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.250194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.250420 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.250265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.250420 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.250333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-kube-api-access-flcdx\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.250420 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.250364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.251421 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.251395 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.254554 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.254503 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.254676 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.254615 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.256579 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.256554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.258195 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.258134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c60356f-e050-4ddf-b82b-6298b88decca-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.263384 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.263157 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-kube-api-access-flcdx\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.265362 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.265321 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-blt8h\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.420154 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.419997 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:20.463611 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.463094 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" event={"ID":"b1c44826-c266-422d-94d6-658e0247af81","Type":"ContainerStarted","Data":"011e2b31197ad56219340dbf9c532e739c1dd3cc4a21b4800d1190932195c2fd"} Aug 01 00:43:20.601100 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:20.601071 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h"] Aug 01 00:43:20.603047 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:43:20.603017 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c60356f_e050_4ddf_b82b_6298b88decca.slice/crio-271e6843924a872deb00a6fb487ad0e245a375d219f8b64d72811802d2c499a0 WatchSource:0}: Error finding container 271e6843924a872deb00a6fb487ad0e245a375d219f8b64d72811802d2c499a0: Status 404 returned error can't find the container with id 271e6843924a872deb00a6fb487ad0e245a375d219f8b64d72811802d2c499a0 Aug 01 00:43:21.472723 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:21.472688 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" event={"ID":"1c60356f-e050-4ddf-b82b-6298b88decca","Type":"ContainerStarted","Data":"271e6843924a872deb00a6fb487ad0e245a375d219f8b64d72811802d2c499a0"} Aug 01 00:43:22.479181 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:22.479148 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" event={"ID":"b1c44826-c266-422d-94d6-658e0247af81","Type":"ContainerStarted","Data":"2148c1a6a239ebb6d0c6b723f9ee614722301cc399ea064754d28b51ab8933b2"} Aug 01 00:43:22.495928 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:22.495884 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7798dfdbd7-m5lph" podStartSLOduration=1.6248374220000001 podStartE2EDuration="4.495864869s" totalImagesPullingTime="2.871027447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:22.495539911 +0000 UTC m=+580.705168368" watchObservedRunningTime="2026-08-01 00:43:22.495864869 +0000 UTC m=+580.705493326" Aug 01 00:43:23.609912 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.609877 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn"] Aug 01 00:43:23.613587 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.613561 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf"] Aug 01 00:43:23.613738 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.613696 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:23.617839 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.617816 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-n67zk\"" Aug 01 00:43:23.617953 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.617887 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Aug 01 00:43:23.618306 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.618283 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:23.620568 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.620546 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Aug 01 00:43:23.620864 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.620845 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mjwlq\"" Aug 01 00:43:23.625492 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.625368 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn"] Aug 01 00:43:23.631002 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.630980 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf"] Aug 01 00:43:23.684114 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.684082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwgf\" (UniqueName: \"kubernetes.io/projected/3e554008-4f64-43bb-9406-2677b287e547-kube-api-access-ldwgf\") pod \"llmisvc-controller-manager-777c6bd7d5-hpjdf\" (UID: \"3e554008-4f64-43bb-9406-2677b287e547\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:23.684114 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.684115 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftd45\" (UniqueName: \"kubernetes.io/projected/62a42240-2659-4035-aa15-78bc1d79c7b8-kube-api-access-ftd45\") pod \"kserve-controller-manager-d6f4b65b8-fd6cn\" (UID: \"62a42240-2659-4035-aa15-78bc1d79c7b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:23.684382 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.684139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62a42240-2659-4035-aa15-78bc1d79c7b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-fd6cn\" (UID: \"62a42240-2659-4035-aa15-78bc1d79c7b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:23.684382 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.684283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e554008-4f64-43bb-9406-2677b287e547-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hpjdf\" (UID: \"3e554008-4f64-43bb-9406-2677b287e547\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:23.784976 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.784944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62a42240-2659-4035-aa15-78bc1d79c7b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-fd6cn\" (UID: \"62a42240-2659-4035-aa15-78bc1d79c7b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:23.785175 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.785052 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e554008-4f64-43bb-9406-2677b287e547-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hpjdf\" (UID: \"3e554008-4f64-43bb-9406-2677b287e547\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:23.785175 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.785091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwgf\" (UniqueName: \"kubernetes.io/projected/3e554008-4f64-43bb-9406-2677b287e547-kube-api-access-ldwgf\") pod \"llmisvc-controller-manager-777c6bd7d5-hpjdf\" (UID: \"3e554008-4f64-43bb-9406-2677b287e547\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:23.785175 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.785115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftd45\" (UniqueName: \"kubernetes.io/projected/62a42240-2659-4035-aa15-78bc1d79c7b8-kube-api-access-ftd45\") pod \"kserve-controller-manager-d6f4b65b8-fd6cn\" (UID: \"62a42240-2659-4035-aa15-78bc1d79c7b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:23.785376 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:43:23.785229 2568 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Aug 01 00:43:23.785376 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:43:23.785315 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e554008-4f64-43bb-9406-2677b287e547-cert podName:3e554008-4f64-43bb-9406-2677b287e547 nodeName:}" failed. No retries permitted until 2026-08-01 00:43:24.285295036 +0000 UTC m=+582.494923473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3e554008-4f64-43bb-9406-2677b287e547-cert") pod "llmisvc-controller-manager-777c6bd7d5-hpjdf" (UID: "3e554008-4f64-43bb-9406-2677b287e547") : secret "llmisvc-webhook-server-cert" not found Aug 01 00:43:23.788191 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.788159 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62a42240-2659-4035-aa15-78bc1d79c7b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-fd6cn\" (UID: \"62a42240-2659-4035-aa15-78bc1d79c7b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:23.796438 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.796413 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftd45\" (UniqueName: \"kubernetes.io/projected/62a42240-2659-4035-aa15-78bc1d79c7b8-kube-api-access-ftd45\") pod \"kserve-controller-manager-d6f4b65b8-fd6cn\" (UID: \"62a42240-2659-4035-aa15-78bc1d79c7b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:23.807591 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.807570 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwgf\" (UniqueName: \"kubernetes.io/projected/3e554008-4f64-43bb-9406-2677b287e547-kube-api-access-ldwgf\") pod \"llmisvc-controller-manager-777c6bd7d5-hpjdf\" (UID: \"3e554008-4f64-43bb-9406-2677b287e547\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:23.929835 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:23.929806 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:24.086835 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:24.086807 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn"] Aug 01 00:43:24.293128 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:24.293039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e554008-4f64-43bb-9406-2677b287e547-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hpjdf\" (UID: \"3e554008-4f64-43bb-9406-2677b287e547\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:24.295942 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:24.295906 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e554008-4f64-43bb-9406-2677b287e547-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hpjdf\" (UID: \"3e554008-4f64-43bb-9406-2677b287e547\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:24.538018 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:24.537985 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:24.904873 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:43:24.904798 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a42240_2659_4035_aa15_78bc1d79c7b8.slice/crio-055a06ed967d77386ebba95fc74cea4e5f3d0e23378d4c0199fb9feb77051d99 WatchSource:0}: Error finding container 055a06ed967d77386ebba95fc74cea4e5f3d0e23378d4c0199fb9feb77051d99: Status 404 returned error can't find the container with id 055a06ed967d77386ebba95fc74cea4e5f3d0e23378d4c0199fb9feb77051d99 Aug 01 00:43:24.985067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:24.985027 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:43:24.985178 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:24.985098 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:43:25.076979 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:25.076955 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf"] Aug 01 00:43:25.078928 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:43:25.078902 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e554008_4f64_43bb_9406_2677b287e547.slice/crio-9f47b1befcdd43df0f1eb5fa6a07b8aedb80481657cc4964aa3b6173bccd4797 WatchSource:0}: Error finding container 9f47b1befcdd43df0f1eb5fa6a07b8aedb80481657cc4964aa3b6173bccd4797: Status 404 returned error can't find the container with id 9f47b1befcdd43df0f1eb5fa6a07b8aedb80481657cc4964aa3b6173bccd4797 Aug 01 00:43:25.500448 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:25.500395 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" event={"ID":"1c60356f-e050-4ddf-b82b-6298b88decca","Type":"ContainerStarted","Data":"e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38"} Aug 01 00:43:25.500730 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:25.500526 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:25.501795 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:25.501754 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" event={"ID":"3e554008-4f64-43bb-9406-2677b287e547","Type":"ContainerStarted","Data":"9f47b1befcdd43df0f1eb5fa6a07b8aedb80481657cc4964aa3b6173bccd4797"} Aug 01 00:43:25.503121 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:25.503100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" event={"ID":"62a42240-2659-4035-aa15-78bc1d79c7b8","Type":"ContainerStarted","Data":"055a06ed967d77386ebba95fc74cea4e5f3d0e23378d4c0199fb9feb77051d99"} Aug 01 00:43:25.525180 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:25.525138 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" podStartSLOduration=1.145916581 podStartE2EDuration="5.525123717s" totalImagesPullingTime="4.379207136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:25.522991224 +0000 UTC m=+583.732619684" watchObservedRunningTime="2026-08-01 00:43:25.525123717 +0000 UTC m=+583.734752175" Aug 01 00:43:26.266464 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.266436 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv"] Aug 01 00:43:26.270363 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.270338 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.272543 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.272519 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4slcr\"" Aug 01 00:43:26.281067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.281039 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv"] Aug 01 00:43:26.423453 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.423629 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.423629 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423507 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.423629 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.423629 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423556 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldg7x\" (UniqueName: \"kubernetes.io/projected/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-kube-api-access-ldg7x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.423629 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423583 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.423911 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423644 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.423911 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.423911 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.423687 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.513823 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.513792 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:26.524803 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.524722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.524803 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.524772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525033 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.524808 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525033 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.524866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525033 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.524915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525033 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.524942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525033 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.524972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525033 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.525015 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldg7x\" (UniqueName: \"kubernetes.io/projected/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-kube-api-access-ldg7x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525596 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.525048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525596 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.525133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.525928 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.525718 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.526007 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.525925 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.526007 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.525931 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.526582 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.526533 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.529523 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.529468 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.530902 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.530878 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.540126 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.540100 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.540799 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.540770 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldg7x\" (UniqueName: \"kubernetes.io/projected/2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f-kube-api-access-ldg7x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv\" (UID: \"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:26.584500 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:26.584460 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:27.403862 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:27.403832 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h"] Aug 01 00:43:27.710246 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:27.710139 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv"] Aug 01 00:43:27.718187 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:43:27.718153 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e00ef10_1dd8_4ac3_a5ca_7aefbaeb190f.slice/crio-ff72a3fa5bb8e402786cf7cf27b223a5b91359de035e611f7fd54d378274716c WatchSource:0}: Error finding container ff72a3fa5bb8e402786cf7cf27b223a5b91359de035e611f7fd54d378274716c: Status 404 returned error can't find the container with id ff72a3fa5bb8e402786cf7cf27b223a5b91359de035e611f7fd54d378274716c Aug 01 00:43:28.522116 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.522079 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" event={"ID":"62a42240-2659-4035-aa15-78bc1d79c7b8","Type":"ContainerStarted","Data":"4b301841fcb8c4e60f46b8c938c9af8e5b355215c17253435032a10488bbd79c"} Aug 01 00:43:28.522612 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.522147 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:43:28.523492 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.523461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" event={"ID":"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f","Type":"ContainerStarted","Data":"ff72a3fa5bb8e402786cf7cf27b223a5b91359de035e611f7fd54d378274716c"} Aug 01 00:43:28.524930 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.524904 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" event={"ID":"3e554008-4f64-43bb-9406-2677b287e547","Type":"ContainerStarted","Data":"b81a4320191f32437d70e832decfdfd1652a2acbad9b22671591ba77e538a497"} Aug 01 00:43:28.525065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.524990 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" podUID="1c60356f-e050-4ddf-b82b-6298b88decca" containerName="discovery" containerID="cri-o://e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38" gracePeriod=30 Aug 01 00:43:28.540515 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.540470 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" podStartSLOduration=2.78415322 podStartE2EDuration="5.540454445s" totalImagesPullingTime="2.756301225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:28.538136335 +0000 UTC m=+586.747764795" watchObservedRunningTime="2026-08-01 00:43:28.540454445 +0000 UTC m=+586.750082902" Aug 01 00:43:28.572302 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.572245 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" podStartSLOduration=2.297737118 podStartE2EDuration="5.572206869s" totalImagesPullingTime="3.274469751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:28.570118301 +0000 UTC m=+586.779746759" watchObservedRunningTime="2026-08-01 00:43:28.572206869 +0000 UTC m=+586.781835328" Aug 01 00:43:28.786567 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.786538 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:28.956135 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.956104 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-cacerts\") pod \"1c60356f-e050-4ddf-b82b-6298b88decca\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " Aug 01 00:43:28.956357 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.956181 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-dns-cert\") pod \"1c60356f-e050-4ddf-b82b-6298b88decca\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " Aug 01 00:43:28.956357 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.956268 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c60356f-e050-4ddf-b82b-6298b88decca-local-certs\") pod \"1c60356f-e050-4ddf-b82b-6298b88decca\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " Aug 01 00:43:28.956357 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.956311 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-ca-configmap\") pod \"1c60356f-e050-4ddf-b82b-6298b88decca\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " Aug 01 00:43:28.956555 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.956364 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-kube-api-access-flcdx\") pod \"1c60356f-e050-4ddf-b82b-6298b88decca\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " Aug 01 00:43:28.956555 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.956398 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-istio-token\") pod \"1c60356f-e050-4ddf-b82b-6298b88decca\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " Aug 01 00:43:28.956555 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.956424 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-kubeconfig\") pod \"1c60356f-e050-4ddf-b82b-6298b88decca\" (UID: \"1c60356f-e050-4ddf-b82b-6298b88decca\") " Aug 01 00:43:28.956835 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.956806 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1c60356f-e050-4ddf-b82b-6298b88decca" (UID: "1c60356f-e050-4ddf-b82b-6298b88decca"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 00:43:28.959681 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.959641 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-cacerts" (OuterVolumeSpecName: "cacerts") pod "1c60356f-e050-4ddf-b82b-6298b88decca" (UID: "1c60356f-e050-4ddf-b82b-6298b88decca"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:43:28.959955 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.959923 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-kube-api-access-flcdx" (OuterVolumeSpecName: "kube-api-access-flcdx") pod "1c60356f-e050-4ddf-b82b-6298b88decca" (UID: "1c60356f-e050-4ddf-b82b-6298b88decca"). InnerVolumeSpecName "kube-api-access-flcdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:43:28.960046 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.959985 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1c60356f-e050-4ddf-b82b-6298b88decca" (UID: "1c60356f-e050-4ddf-b82b-6298b88decca"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:43:28.960196 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.960172 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c60356f-e050-4ddf-b82b-6298b88decca" (UID: "1c60356f-e050-4ddf-b82b-6298b88decca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:43:28.960653 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.960620 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c60356f-e050-4ddf-b82b-6298b88decca-local-certs" (OuterVolumeSpecName: "local-certs") pod "1c60356f-e050-4ddf-b82b-6298b88decca" (UID: "1c60356f-e050-4ddf-b82b-6298b88decca"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:43:28.960901 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:28.960880 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1c60356f-e050-4ddf-b82b-6298b88decca" (UID: "1c60356f-e050-4ddf-b82b-6298b88decca"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:43:29.057834 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.057793 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-dns-cert\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:43:29.057834 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.057833 2568 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c60356f-e050-4ddf-b82b-6298b88decca-local-certs\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:43:29.058068 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.057846 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c60356f-e050-4ddf-b82b-6298b88decca-istio-csr-ca-configmap\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:43:29.058068 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.057860 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flcdx\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-kube-api-access-flcdx\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:43:29.058068 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.057872 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c60356f-e050-4ddf-b82b-6298b88decca-istio-token\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:43:29.058068 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.057882 2568 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-istio-kubeconfig\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:43:29.058068 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.057892 2568 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c60356f-e050-4ddf-b82b-6298b88decca-cacerts\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:43:29.457615 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.457581 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-6vffs" Aug 01 00:43:29.533116 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.533084 2568 generic.go:356] "Generic (PLEG): container finished" podID="1c60356f-e050-4ddf-b82b-6298b88decca" containerID="e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38" exitCode=0 Aug 01 00:43:29.533575 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.533141 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" Aug 01 00:43:29.533575 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.533162 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" event={"ID":"1c60356f-e050-4ddf-b82b-6298b88decca","Type":"ContainerDied","Data":"e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38"} Aug 01 00:43:29.533575 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.533244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h" event={"ID":"1c60356f-e050-4ddf-b82b-6298b88decca","Type":"ContainerDied","Data":"271e6843924a872deb00a6fb487ad0e245a375d219f8b64d72811802d2c499a0"} Aug 01 00:43:29.533575 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.533277 2568 scope.go:117] "RemoveContainer" containerID="e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38" Aug 01 00:43:29.534050 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.534028 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:43:29.546170 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.546151 2568 scope.go:117] "RemoveContainer" containerID="e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38" Aug 01 00:43:29.546518 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:43:29.546494 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38\": container with ID starting with e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38 not found: ID does not exist" containerID="e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38" Aug 01 00:43:29.546609 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.546530 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38"} err="failed to get container status \"e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38\": rpc error: code = NotFound desc = could not find container \"e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38\": container with ID starting with e34a99df5efb945283796d912fe468d9623658a1dfbeb4ec2121bd303af07c38 not found: ID does not exist" Aug 01 00:43:29.558601 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.558573 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h"] Aug 01 00:43:29.562660 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:29.562637 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-blt8h"] Aug 01 00:43:30.232282 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:30.232249 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c60356f-e050-4ddf-b82b-6298b88decca" path="/var/lib/kubelet/pods/1c60356f-e050-4ddf-b82b-6298b88decca/volumes" Aug 01 00:43:31.091954 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:31.091921 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:43:31.092240 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:31.091990 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:43:31.092240 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:31.092021 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:43:31.547539 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:31.547502 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" event={"ID":"2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f","Type":"ContainerStarted","Data":"da56b59ac2a4184d753c7d65461d72957fcf6ec8f80d473066857521ef9b1169"} Aug 01 00:43:31.566958 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:31.566908 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" podStartSLOduration=2.195861498 podStartE2EDuration="5.566891769s" totalImagesPullingTime="3.371030271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:31.565323937 +0000 UTC m=+589.774952393" watchObservedRunningTime="2026-08-01 00:43:31.566891769 +0000 UTC m=+589.776520225" Aug 01 00:43:31.585203 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:31.585171 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:32.590503 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:32.590470 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:33.558570 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:33.558539 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:33.559484 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:33.559461 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv" Aug 01 00:43:46.932049 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.932017 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db46f485b-kmh86"] Aug 01 00:43:46.934501 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.932416 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c60356f-e050-4ddf-b82b-6298b88decca" containerName="discovery" Aug 01 00:43:46.934501 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.932427 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c60356f-e050-4ddf-b82b-6298b88decca" containerName="discovery" Aug 01 00:43:46.934501 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.932484 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c60356f-e050-4ddf-b82b-6298b88decca" containerName="discovery" Aug 01 00:43:46.935396 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.935381 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:46.937641 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.937614 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 01 00:43:46.937792 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.937660 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 01 00:43:46.937792 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.937681 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 01 00:43:46.938568 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.938536 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 01 00:43:46.938746 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.938568 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 01 00:43:46.938746 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.938568 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8tgrd\"" Aug 01 00:43:46.944182 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.944160 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 01 00:43:46.950647 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:46.950289 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db46f485b-kmh86"] Aug 01 00:43:47.015077 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.015029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-serving-cert\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.015077 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.015080 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-trusted-ca-bundle\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.015314 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.015130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-config\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.015314 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.015147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-oauth-serving-cert\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.015314 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.015196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-oauth-config\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.015314 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.015240 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdc62\" (UniqueName: \"kubernetes.io/projected/01116a7c-9330-4c8c-82d4-b4947ae0c42a-kube-api-access-vdc62\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.015314 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.015310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-service-ca\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.116060 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-service-ca\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.116207 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-serving-cert\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.116207 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-trusted-ca-bundle\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.116207 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-config\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.116207 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-oauth-serving-cert\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.116424 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-oauth-config\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.116424 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdc62\" (UniqueName: \"kubernetes.io/projected/01116a7c-9330-4c8c-82d4-b4947ae0c42a-kube-api-access-vdc62\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.116830 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116802 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-service-ca\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.117106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.117028 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-oauth-serving-cert\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.117106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.116976 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-config\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.117247 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.117037 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01116a7c-9330-4c8c-82d4-b4947ae0c42a-trusted-ca-bundle\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.118772 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.118743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-oauth-config\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.118956 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.118934 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01116a7c-9330-4c8c-82d4-b4947ae0c42a-console-serving-cert\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.136525 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.136501 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdc62\" (UniqueName: \"kubernetes.io/projected/01116a7c-9330-4c8c-82d4-b4947ae0c42a-kube-api-access-vdc62\") pod \"console-5db46f485b-kmh86\" (UID: \"01116a7c-9330-4c8c-82d4-b4947ae0c42a\") " pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.255939 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.255855 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:47.388149 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.388121 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db46f485b-kmh86"] Aug 01 00:43:47.390437 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:43:47.390409 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01116a7c_9330_4c8c_82d4_b4947ae0c42a.slice/crio-0ce7155a9b2ce6c55f08f2e466271f8bb3bb2e8a717654f6b290a8411c090fd5 WatchSource:0}: Error finding container 0ce7155a9b2ce6c55f08f2e466271f8bb3bb2e8a717654f6b290a8411c090fd5: Status 404 returned error can't find the container with id 0ce7155a9b2ce6c55f08f2e466271f8bb3bb2e8a717654f6b290a8411c090fd5 Aug 01 00:43:47.635619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.635524 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db46f485b-kmh86" event={"ID":"01116a7c-9330-4c8c-82d4-b4947ae0c42a","Type":"ContainerStarted","Data":"e75b73691d3731825248881eddaed8a6200cb3752c3264f8e5655ddb3d02b242"} Aug 01 00:43:47.635619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.635576 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db46f485b-kmh86" event={"ID":"01116a7c-9330-4c8c-82d4-b4947ae0c42a","Type":"ContainerStarted","Data":"0ce7155a9b2ce6c55f08f2e466271f8bb3bb2e8a717654f6b290a8411c090fd5"} Aug 01 00:43:47.652658 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:47.652608 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5db46f485b-kmh86" podStartSLOduration=1.652593489 podStartE2EDuration="1.652593489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:47.651067766 +0000 UTC m=+605.860696223" watchObservedRunningTime="2026-08-01 00:43:47.652593489 +0000 UTC m=+605.862221946" Aug 01 00:43:57.256173 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:57.256136 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:57.256676 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:57.256197 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:57.261076 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:57.261050 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:57.690975 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:57.690945 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5db46f485b-kmh86" Aug 01 00:43:59.538537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:43:59.538500 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fd6cn" Aug 01 00:44:00.543669 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:00.543637 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpjdf" Aug 01 00:44:17.685180 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.685140 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474"] Aug 01 00:44:17.688866 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.688843 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.691106 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.691087 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 01 00:44:17.691244 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.691087 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 01 00:44:17.691244 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.691129 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4pnxs\"" Aug 01 00:44:17.696574 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.696551 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474"] Aug 01 00:44:17.783853 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.783820 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv"] Aug 01 00:44:17.783853 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.783839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.784069 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.783880 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.784069 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.783947 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlszl\" (UniqueName: \"kubernetes.io/projected/18b057e5-f2f2-43b9-9660-fde1603297df-kube-api-access-wlszl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.787725 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.787704 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.795744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.795717 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv"] Aug 01 00:44:17.882441 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.882404 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z"] Aug 01 00:44:17.885323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.885301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.885431 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.885342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.885431 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.885376 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.885510 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.885486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlszl\" (UniqueName: \"kubernetes.io/projected/18b057e5-f2f2-43b9-9660-fde1603297df-kube-api-access-wlszl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.885544 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.885532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.885621 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.885603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92ns\" (UniqueName: \"kubernetes.io/projected/ab0450d2-eeab-4086-b935-580022508f8a-kube-api-access-v92ns\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.885740 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.885720 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.885740 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.885734 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.886264 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.886250 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:17.894459 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.894434 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z"] Aug 01 00:44:17.896246 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.896200 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlszl\" (UniqueName: \"kubernetes.io/projected/18b057e5-f2f2-43b9-9660-fde1603297df-kube-api-access-wlszl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:17.982808 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.982733 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49"] Aug 01 00:44:17.986674 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.986654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:17.986796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.986702 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.986796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.986749 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.986796 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.986778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf2qw\" (UniqueName: \"kubernetes.io/projected/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-kube-api-access-sf2qw\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:17.986959 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.986809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:17.986959 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.986833 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:17.986959 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.986857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92ns\" (UniqueName: \"kubernetes.io/projected/ab0450d2-eeab-4086-b935-580022508f8a-kube-api-access-v92ns\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.987129 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.987110 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.987190 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.987153 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.995414 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.995393 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49"] Aug 01 00:44:17.997082 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.997061 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92ns\" (UniqueName: \"kubernetes.io/projected/ab0450d2-eeab-4086-b935-580022508f8a-kube-api-access-v92ns\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:17.999869 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:17.999847 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:18.087933 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.087894 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf2qw\" (UniqueName: \"kubernetes.io/projected/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-kube-api-access-sf2qw\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:18.088067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.087960 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2msxv\" (UniqueName: \"kubernetes.io/projected/bba21c48-eca1-4965-8885-fc27de4f7309-kube-api-access-2msxv\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.088067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.087990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:18.088067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.088022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.088067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.088053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:18.088356 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.088185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.088538 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.088502 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:18.088619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.088589 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:18.096327 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.096299 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf2qw\" (UniqueName: \"kubernetes.io/projected/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-kube-api-access-sf2qw\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:18.097931 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.097905 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:18.137364 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.137335 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474"] Aug 01 00:44:18.139049 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:18.139024 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b057e5_f2f2_43b9_9660_fde1603297df.slice/crio-be4d6f47a73ea56d0b45d5a5bd59915ecdbe25bebf426f9017f80d25396a57ae WatchSource:0}: Error finding container be4d6f47a73ea56d0b45d5a5bd59915ecdbe25bebf426f9017f80d25396a57ae: Status 404 returned error can't find the container with id be4d6f47a73ea56d0b45d5a5bd59915ecdbe25bebf426f9017f80d25396a57ae Aug 01 00:44:18.189131 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.189101 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.189309 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.189161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2msxv\" (UniqueName: \"kubernetes.io/projected/bba21c48-eca1-4965-8885-fc27de4f7309-kube-api-access-2msxv\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.189309 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.189194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.189482 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.189463 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.189546 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.189532 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.197510 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.197318 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:18.198072 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.198047 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2msxv\" (UniqueName: \"kubernetes.io/projected/bba21c48-eca1-4965-8885-fc27de4f7309-kube-api-access-2msxv\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.252511 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.252478 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv"] Aug 01 00:44:18.253490 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:18.253457 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0450d2_eeab_4086_b935_580022508f8a.slice/crio-1510f21a88b709a4b8613e5ce7412814e63686f751916cd9dab21b1be26befa1 WatchSource:0}: Error finding container 1510f21a88b709a4b8613e5ce7412814e63686f751916cd9dab21b1be26befa1: Status 404 returned error can't find the container with id 1510f21a88b709a4b8613e5ce7412814e63686f751916cd9dab21b1be26befa1 Aug 01 00:44:18.296631 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.296599 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:18.356597 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.355315 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z"] Aug 01 00:44:18.362673 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:18.362522 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c454c0e_0ffc_4a8e_ba26_61e4f3139909.slice/crio-b2d1dcd5607ae92497195db4efc222dbbfcd69e5add15daa28822afbf3604535 WatchSource:0}: Error finding container b2d1dcd5607ae92497195db4efc222dbbfcd69e5add15daa28822afbf3604535: Status 404 returned error can't find the container with id b2d1dcd5607ae92497195db4efc222dbbfcd69e5add15daa28822afbf3604535 Aug 01 00:44:18.453038 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.452998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49"] Aug 01 00:44:18.467327 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:18.467291 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba21c48_eca1_4965_8885_fc27de4f7309.slice/crio-399fc4574b3dc6cc82b0bc0a97cbb63b7aed5ad4904126740a93774bbcface73 WatchSource:0}: Error finding container 399fc4574b3dc6cc82b0bc0a97cbb63b7aed5ad4904126740a93774bbcface73: Status 404 returned error can't find the container with id 399fc4574b3dc6cc82b0bc0a97cbb63b7aed5ad4904126740a93774bbcface73 Aug 01 00:44:18.794376 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.794300 2568 generic.go:356] "Generic (PLEG): container finished" podID="bba21c48-eca1-4965-8885-fc27de4f7309" containerID="b8caaeb316ca50d8fb10587bf8fcde07ccc9af73c69a87faefe2498fcb1e1884" exitCode=0 Aug 01 00:44:18.794813 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.794375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" event={"ID":"bba21c48-eca1-4965-8885-fc27de4f7309","Type":"ContainerDied","Data":"b8caaeb316ca50d8fb10587bf8fcde07ccc9af73c69a87faefe2498fcb1e1884"} Aug 01 00:44:18.794813 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.794417 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" event={"ID":"bba21c48-eca1-4965-8885-fc27de4f7309","Type":"ContainerStarted","Data":"399fc4574b3dc6cc82b0bc0a97cbb63b7aed5ad4904126740a93774bbcface73"} Aug 01 00:44:18.796014 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.795990 2568 generic.go:356] "Generic (PLEG): container finished" podID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerID="5f5b71047882e158a9369d38f3a9c716a5e089bbca9cdd75e4975f8106be4d6f" exitCode=0 Aug 01 00:44:18.796105 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.796082 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" event={"ID":"0c454c0e-0ffc-4a8e-ba26-61e4f3139909","Type":"ContainerDied","Data":"5f5b71047882e158a9369d38f3a9c716a5e089bbca9cdd75e4975f8106be4d6f"} Aug 01 00:44:18.796176 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.796132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" event={"ID":"0c454c0e-0ffc-4a8e-ba26-61e4f3139909","Type":"ContainerStarted","Data":"b2d1dcd5607ae92497195db4efc222dbbfcd69e5add15daa28822afbf3604535"} Aug 01 00:44:18.797680 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.797657 2568 generic.go:356] "Generic (PLEG): container finished" podID="ab0450d2-eeab-4086-b935-580022508f8a" containerID="5bc4b43951ed5f672e5a174618c9e58afad79ef574e8dab93677f182f9f9462c" exitCode=0 Aug 01 00:44:18.797763 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.797742 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" event={"ID":"ab0450d2-eeab-4086-b935-580022508f8a","Type":"ContainerDied","Data":"5bc4b43951ed5f672e5a174618c9e58afad79ef574e8dab93677f182f9f9462c"} Aug 01 00:44:18.797828 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.797788 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" event={"ID":"ab0450d2-eeab-4086-b935-580022508f8a","Type":"ContainerStarted","Data":"1510f21a88b709a4b8613e5ce7412814e63686f751916cd9dab21b1be26befa1"} Aug 01 00:44:18.799352 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.799332 2568 generic.go:356] "Generic (PLEG): container finished" podID="18b057e5-f2f2-43b9-9660-fde1603297df" containerID="c1b086ae093b0e28598ab28920f3e964110915b0be7c541c06a9125ba7c9f31f" exitCode=0 Aug 01 00:44:18.799425 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.799406 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" event={"ID":"18b057e5-f2f2-43b9-9660-fde1603297df","Type":"ContainerDied","Data":"c1b086ae093b0e28598ab28920f3e964110915b0be7c541c06a9125ba7c9f31f"} Aug 01 00:44:18.799471 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:18.799447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" event={"ID":"18b057e5-f2f2-43b9-9660-fde1603297df","Type":"ContainerStarted","Data":"be4d6f47a73ea56d0b45d5a5bd59915ecdbe25bebf426f9017f80d25396a57ae"} Aug 01 00:44:19.813820 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:19.813784 2568 generic.go:356] "Generic (PLEG): container finished" podID="bba21c48-eca1-4965-8885-fc27de4f7309" containerID="259fbef18050d6c6c185d13eac9412661e08a112e7e40917f46cb6b5c034b1c1" exitCode=0 Aug 01 00:44:19.814180 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:19.813860 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" event={"ID":"bba21c48-eca1-4965-8885-fc27de4f7309","Type":"ContainerDied","Data":"259fbef18050d6c6c185d13eac9412661e08a112e7e40917f46cb6b5c034b1c1"} Aug 01 00:44:20.821369 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:20.821333 2568 generic.go:356] "Generic (PLEG): container finished" podID="bba21c48-eca1-4965-8885-fc27de4f7309" containerID="fbe95f8589d49cef7331319107098783e2a73b0aacf31113771cff1d61b6f98a" exitCode=0 Aug 01 00:44:20.821780 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:20.821416 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" event={"ID":"bba21c48-eca1-4965-8885-fc27de4f7309","Type":"ContainerDied","Data":"fbe95f8589d49cef7331319107098783e2a73b0aacf31113771cff1d61b6f98a"} Aug 01 00:44:20.823048 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:20.823026 2568 generic.go:356] "Generic (PLEG): container finished" podID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerID="c2e70c1bb369b24563ac208c3ab19ca4a0717d7faebbd776abcce6b9c25fabf6" exitCode=0 Aug 01 00:44:20.823166 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:20.823088 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" event={"ID":"0c454c0e-0ffc-4a8e-ba26-61e4f3139909","Type":"ContainerDied","Data":"c2e70c1bb369b24563ac208c3ab19ca4a0717d7faebbd776abcce6b9c25fabf6"} Aug 01 00:44:20.825120 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:20.825098 2568 generic.go:356] "Generic (PLEG): container finished" podID="ab0450d2-eeab-4086-b935-580022508f8a" containerID="74c890acf8ec9f769618e6c8101372200a517579ac3f3c76f85422d1ad04802c" exitCode=0 Aug 01 00:44:20.825248 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:20.825180 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" event={"ID":"ab0450d2-eeab-4086-b935-580022508f8a","Type":"ContainerDied","Data":"74c890acf8ec9f769618e6c8101372200a517579ac3f3c76f85422d1ad04802c"} Aug 01 00:44:20.827087 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:20.827067 2568 generic.go:356] "Generic (PLEG): container finished" podID="18b057e5-f2f2-43b9-9660-fde1603297df" containerID="0558f2142277e4c15465fafc8965be0fd222c9513bc18ea74d879d22599c4b6c" exitCode=0 Aug 01 00:44:20.827146 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:20.827123 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" event={"ID":"18b057e5-f2f2-43b9-9660-fde1603297df","Type":"ContainerDied","Data":"0558f2142277e4c15465fafc8965be0fd222c9513bc18ea74d879d22599c4b6c"} Aug 01 00:44:21.834125 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:21.834084 2568 generic.go:356] "Generic (PLEG): container finished" podID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerID="a410b80df0cead1be42f896b4826807c387a9f515780cf4333866b965e0ec8e5" exitCode=0 Aug 01 00:44:21.834559 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:21.834182 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" event={"ID":"0c454c0e-0ffc-4a8e-ba26-61e4f3139909","Type":"ContainerDied","Data":"a410b80df0cead1be42f896b4826807c387a9f515780cf4333866b965e0ec8e5"} Aug 01 00:44:21.839272 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:21.839249 2568 generic.go:356] "Generic (PLEG): container finished" podID="ab0450d2-eeab-4086-b935-580022508f8a" containerID="f3bf2014ef614cad2b0ca19f235ca6a965d5ace7029da08a5f11ce67259f0f24" exitCode=0 Aug 01 00:44:21.839386 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:21.839317 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" event={"ID":"ab0450d2-eeab-4086-b935-580022508f8a","Type":"ContainerDied","Data":"f3bf2014ef614cad2b0ca19f235ca6a965d5ace7029da08a5f11ce67259f0f24"} Aug 01 00:44:21.841602 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:21.841574 2568 generic.go:356] "Generic (PLEG): container finished" podID="18b057e5-f2f2-43b9-9660-fde1603297df" containerID="0c8aa6960b94a940d81f1b1bb3fef67f79ff7a35896c422729b277c666af34f9" exitCode=0 Aug 01 00:44:21.841705 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:21.841659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" event={"ID":"18b057e5-f2f2-43b9-9660-fde1603297df","Type":"ContainerDied","Data":"0c8aa6960b94a940d81f1b1bb3fef67f79ff7a35896c422729b277c666af34f9"} Aug 01 00:44:21.977831 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:21.977809 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:22.124526 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.124436 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-bundle\") pod \"bba21c48-eca1-4965-8885-fc27de4f7309\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " Aug 01 00:44:22.124526 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.124514 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-util\") pod \"bba21c48-eca1-4965-8885-fc27de4f7309\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " Aug 01 00:44:22.124744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.124652 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2msxv\" (UniqueName: \"kubernetes.io/projected/bba21c48-eca1-4965-8885-fc27de4f7309-kube-api-access-2msxv\") pod \"bba21c48-eca1-4965-8885-fc27de4f7309\" (UID: \"bba21c48-eca1-4965-8885-fc27de4f7309\") " Aug 01 00:44:22.124920 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.124888 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-bundle" (OuterVolumeSpecName: "bundle") pod "bba21c48-eca1-4965-8885-fc27de4f7309" (UID: "bba21c48-eca1-4965-8885-fc27de4f7309"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:22.126942 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.126919 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba21c48-eca1-4965-8885-fc27de4f7309-kube-api-access-2msxv" (OuterVolumeSpecName: "kube-api-access-2msxv") pod "bba21c48-eca1-4965-8885-fc27de4f7309" (UID: "bba21c48-eca1-4965-8885-fc27de4f7309"). InnerVolumeSpecName "kube-api-access-2msxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:44:22.131983 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.131940 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-util" (OuterVolumeSpecName: "util") pod "bba21c48-eca1-4965-8885-fc27de4f7309" (UID: "bba21c48-eca1-4965-8885-fc27de4f7309"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:22.225598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.225569 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2msxv\" (UniqueName: \"kubernetes.io/projected/bba21c48-eca1-4965-8885-fc27de4f7309-kube-api-access-2msxv\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:22.225598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.225594 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-bundle\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:22.225751 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.225609 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bba21c48-eca1-4965-8885-fc27de4f7309-util\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:22.849485 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.849457 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" Aug 01 00:44:22.849868 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.849490 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a06lk49" event={"ID":"bba21c48-eca1-4965-8885-fc27de4f7309","Type":"ContainerDied","Data":"399fc4574b3dc6cc82b0bc0a97cbb63b7aed5ad4904126740a93774bbcface73"} Aug 01 00:44:22.849868 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.849563 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="399fc4574b3dc6cc82b0bc0a97cbb63b7aed5ad4904126740a93774bbcface73" Aug 01 00:44:22.995019 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:22.994993 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:23.047712 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.047681 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:23.050456 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.050439 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:23.134906 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.134825 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-util\") pod \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " Aug 01 00:44:23.134906 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.134878 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-bundle\") pod \"ab0450d2-eeab-4086-b935-580022508f8a\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " Aug 01 00:44:23.134906 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.134897 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-bundle\") pod \"18b057e5-f2f2-43b9-9660-fde1603297df\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " Aug 01 00:44:23.135279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.134913 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-bundle\") pod \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " Aug 01 00:44:23.135279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.134952 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-util\") pod \"18b057e5-f2f2-43b9-9660-fde1603297df\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " Aug 01 00:44:23.135279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.134984 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-util\") pod \"ab0450d2-eeab-4086-b935-580022508f8a\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " Aug 01 00:44:23.135279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.135064 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlszl\" (UniqueName: \"kubernetes.io/projected/18b057e5-f2f2-43b9-9660-fde1603297df-kube-api-access-wlszl\") pod \"18b057e5-f2f2-43b9-9660-fde1603297df\" (UID: \"18b057e5-f2f2-43b9-9660-fde1603297df\") " Aug 01 00:44:23.135279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.135091 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v92ns\" (UniqueName: \"kubernetes.io/projected/ab0450d2-eeab-4086-b935-580022508f8a-kube-api-access-v92ns\") pod \"ab0450d2-eeab-4086-b935-580022508f8a\" (UID: \"ab0450d2-eeab-4086-b935-580022508f8a\") " Aug 01 00:44:23.135279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.135119 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf2qw\" (UniqueName: \"kubernetes.io/projected/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-kube-api-access-sf2qw\") pod \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\" (UID: \"0c454c0e-0ffc-4a8e-ba26-61e4f3139909\") " Aug 01 00:44:23.135586 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.135492 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-bundle" (OuterVolumeSpecName: "bundle") pod "0c454c0e-0ffc-4a8e-ba26-61e4f3139909" (UID: "0c454c0e-0ffc-4a8e-ba26-61e4f3139909"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:23.135637 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.135603 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-bundle" (OuterVolumeSpecName: "bundle") pod "ab0450d2-eeab-4086-b935-580022508f8a" (UID: "ab0450d2-eeab-4086-b935-580022508f8a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:23.135753 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.135727 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-bundle" (OuterVolumeSpecName: "bundle") pod "18b057e5-f2f2-43b9-9660-fde1603297df" (UID: "18b057e5-f2f2-43b9-9660-fde1603297df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:23.137697 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.137666 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-kube-api-access-sf2qw" (OuterVolumeSpecName: "kube-api-access-sf2qw") pod "0c454c0e-0ffc-4a8e-ba26-61e4f3139909" (UID: "0c454c0e-0ffc-4a8e-ba26-61e4f3139909"). InnerVolumeSpecName "kube-api-access-sf2qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:44:23.138101 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.138076 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0450d2-eeab-4086-b935-580022508f8a-kube-api-access-v92ns" (OuterVolumeSpecName: "kube-api-access-v92ns") pod "ab0450d2-eeab-4086-b935-580022508f8a" (UID: "ab0450d2-eeab-4086-b935-580022508f8a"). InnerVolumeSpecName "kube-api-access-v92ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:44:23.138321 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.138302 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b057e5-f2f2-43b9-9660-fde1603297df-kube-api-access-wlszl" (OuterVolumeSpecName: "kube-api-access-wlszl") pod "18b057e5-f2f2-43b9-9660-fde1603297df" (UID: "18b057e5-f2f2-43b9-9660-fde1603297df"). InnerVolumeSpecName "kube-api-access-wlszl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:44:23.141485 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.141452 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-util" (OuterVolumeSpecName: "util") pod "18b057e5-f2f2-43b9-9660-fde1603297df" (UID: "18b057e5-f2f2-43b9-9660-fde1603297df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:23.143899 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.143871 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-util" (OuterVolumeSpecName: "util") pod "ab0450d2-eeab-4086-b935-580022508f8a" (UID: "ab0450d2-eeab-4086-b935-580022508f8a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:23.144162 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.144147 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-util" (OuterVolumeSpecName: "util") pod "0c454c0e-0ffc-4a8e-ba26-61e4f3139909" (UID: "0c454c0e-0ffc-4a8e-ba26-61e4f3139909"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:23.236538 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236507 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-util\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.236538 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236532 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-bundle\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.236538 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236541 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-bundle\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.236538 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236548 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-bundle\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.236783 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236555 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18b057e5-f2f2-43b9-9660-fde1603297df-util\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.236783 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236563 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab0450d2-eeab-4086-b935-580022508f8a-util\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.236783 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236573 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlszl\" (UniqueName: \"kubernetes.io/projected/18b057e5-f2f2-43b9-9660-fde1603297df-kube-api-access-wlszl\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.236783 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236584 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v92ns\" (UniqueName: \"kubernetes.io/projected/ab0450d2-eeab-4086-b935-580022508f8a-kube-api-access-v92ns\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.236783 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.236593 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf2qw\" (UniqueName: \"kubernetes.io/projected/0c454c0e-0ffc-4a8e-ba26-61e4f3139909-kube-api-access-sf2qw\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:23.858023 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.857978 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" event={"ID":"0c454c0e-0ffc-4a8e-ba26-61e4f3139909","Type":"ContainerDied","Data":"b2d1dcd5607ae92497195db4efc222dbbfcd69e5add15daa28822afbf3604535"} Aug 01 00:44:23.858471 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.858036 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2d1dcd5607ae92497195db4efc222dbbfcd69e5add15daa28822afbf3604535" Aug 01 00:44:23.858471 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.857985 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6q75z" Aug 01 00:44:23.859764 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.859740 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" Aug 01 00:44:23.859899 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.859740 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c86nhv" event={"ID":"ab0450d2-eeab-4086-b935-580022508f8a","Type":"ContainerDied","Data":"1510f21a88b709a4b8613e5ce7412814e63686f751916cd9dab21b1be26befa1"} Aug 01 00:44:23.859899 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.859872 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1510f21a88b709a4b8613e5ce7412814e63686f751916cd9dab21b1be26befa1" Aug 01 00:44:23.861763 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.861737 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" event={"ID":"18b057e5-f2f2-43b9-9660-fde1603297df","Type":"ContainerDied","Data":"be4d6f47a73ea56d0b45d5a5bd59915ecdbe25bebf426f9017f80d25396a57ae"} Aug 01 00:44:23.861865 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.861781 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be4d6f47a73ea56d0b45d5a5bd59915ecdbe25bebf426f9017f80d25396a57ae" Aug 01 00:44:23.861865 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:23.861788 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wt474" Aug 01 00:44:28.832246 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832203 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd"] Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832565 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerName="pull" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832575 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerName="pull" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832585 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba21c48-eca1-4965-8885-fc27de4f7309" containerName="extract" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832590 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba21c48-eca1-4965-8885-fc27de4f7309" containerName="extract" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832600 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b057e5-f2f2-43b9-9660-fde1603297df" containerName="util" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832605 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b057e5-f2f2-43b9-9660-fde1603297df" containerName="util" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832611 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b057e5-f2f2-43b9-9660-fde1603297df" containerName="pull" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832616 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b057e5-f2f2-43b9-9660-fde1603297df" containerName="pull" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832624 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerName="extract" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832630 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerName="extract" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832640 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0450d2-eeab-4086-b935-580022508f8a" containerName="util" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832645 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0450d2-eeab-4086-b935-580022508f8a" containerName="util" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832651 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerName="util" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832656 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerName="util" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832662 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba21c48-eca1-4965-8885-fc27de4f7309" containerName="pull" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832667 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba21c48-eca1-4965-8885-fc27de4f7309" containerName="pull" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832675 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0450d2-eeab-4086-b935-580022508f8a" containerName="pull" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832680 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0450d2-eeab-4086-b935-580022508f8a" containerName="pull" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832687 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0450d2-eeab-4086-b935-580022508f8a" containerName="extract" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832692 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0450d2-eeab-4086-b935-580022508f8a" containerName="extract" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832698 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b057e5-f2f2-43b9-9660-fde1603297df" containerName="extract" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832703 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b057e5-f2f2-43b9-9660-fde1603297df" containerName="extract" Aug 01 00:44:28.832702 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832710 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba21c48-eca1-4965-8885-fc27de4f7309" containerName="util" Aug 01 00:44:28.833418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832716 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba21c48-eca1-4965-8885-fc27de4f7309" containerName="util" Aug 01 00:44:28.833418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832768 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba21c48-eca1-4965-8885-fc27de4f7309" containerName="extract" Aug 01 00:44:28.833418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832776 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b057e5-f2f2-43b9-9660-fde1603297df" containerName="extract" Aug 01 00:44:28.833418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832785 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab0450d2-eeab-4086-b935-580022508f8a" containerName="extract" Aug 01 00:44:28.833418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.832790 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c454c0e-0ffc-4a8e-ba26-61e4f3139909" containerName="extract" Aug 01 00:44:28.837107 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.837087 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:28.839494 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.839464 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 01 00:44:28.839728 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.839708 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-sl849\"" Aug 01 00:44:28.840200 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.840186 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 01 00:44:28.847504 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.847480 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd"] Aug 01 00:44:28.989756 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:28.989724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbbf\" (UniqueName: \"kubernetes.io/projected/030c88c0-c802-40a3-8d3d-96428d108ff5-kube-api-access-2sbbf\") pod \"limitador-operator-controller-manager-6c549b6d85-qxpsd\" (UID: \"030c88c0-c802-40a3-8d3d-96428d108ff5\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:29.090382 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:29.090289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbbf\" (UniqueName: \"kubernetes.io/projected/030c88c0-c802-40a3-8d3d-96428d108ff5-kube-api-access-2sbbf\") pod \"limitador-operator-controller-manager-6c549b6d85-qxpsd\" (UID: \"030c88c0-c802-40a3-8d3d-96428d108ff5\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:29.103833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:29.103806 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbbf\" (UniqueName: \"kubernetes.io/projected/030c88c0-c802-40a3-8d3d-96428d108ff5-kube-api-access-2sbbf\") pod \"limitador-operator-controller-manager-6c549b6d85-qxpsd\" (UID: \"030c88c0-c802-40a3-8d3d-96428d108ff5\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:29.148494 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:29.148468 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:29.287498 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:29.287432 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd"] Aug 01 00:44:29.289717 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:29.289687 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030c88c0_c802_40a3_8d3d_96428d108ff5.slice/crio-bf8e29ce614fba8c35c48a21c82da59bd9f8916f5e54d5505faf5d77a615d814 WatchSource:0}: Error finding container bf8e29ce614fba8c35c48a21c82da59bd9f8916f5e54d5505faf5d77a615d814: Status 404 returned error can't find the container with id bf8e29ce614fba8c35c48a21c82da59bd9f8916f5e54d5505faf5d77a615d814 Aug 01 00:44:29.895677 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:29.895642 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" event={"ID":"030c88c0-c802-40a3-8d3d-96428d108ff5","Type":"ContainerStarted","Data":"bf8e29ce614fba8c35c48a21c82da59bd9f8916f5e54d5505faf5d77a615d814"} Aug 01 00:44:30.940914 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:30.940882 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc"] Aug 01 00:44:30.944321 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:30.944299 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" Aug 01 00:44:30.946718 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:30.946693 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cxmrm\"" Aug 01 00:44:30.946848 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:30.946736 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 01 00:44:30.965784 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:30.965761 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc"] Aug 01 00:44:31.111292 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:31.111243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpln6\" (UniqueName: \"kubernetes.io/projected/52712577-618b-4f22-ba81-b3b515bbd2a3-kube-api-access-hpln6\") pod \"dns-operator-controller-manager-5985564776-zqsrc\" (UID: \"52712577-618b-4f22-ba81-b3b515bbd2a3\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" Aug 01 00:44:31.212271 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:31.212182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpln6\" (UniqueName: \"kubernetes.io/projected/52712577-618b-4f22-ba81-b3b515bbd2a3-kube-api-access-hpln6\") pod \"dns-operator-controller-manager-5985564776-zqsrc\" (UID: \"52712577-618b-4f22-ba81-b3b515bbd2a3\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" Aug 01 00:44:31.221272 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:31.221246 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpln6\" (UniqueName: \"kubernetes.io/projected/52712577-618b-4f22-ba81-b3b515bbd2a3-kube-api-access-hpln6\") pod \"dns-operator-controller-manager-5985564776-zqsrc\" (UID: \"52712577-618b-4f22-ba81-b3b515bbd2a3\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" Aug 01 00:44:31.255009 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:31.254978 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" Aug 01 00:44:32.234924 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:32.234893 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52712577_618b_4f22_ba81_b3b515bbd2a3.slice/crio-04937afc1c6e8412db3fefe2bdd83000221e0daccd456b737dc2735679f926dd WatchSource:0}: Error finding container 04937afc1c6e8412db3fefe2bdd83000221e0daccd456b737dc2735679f926dd: Status 404 returned error can't find the container with id 04937afc1c6e8412db3fefe2bdd83000221e0daccd456b737dc2735679f926dd Aug 01 00:44:32.237205 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:32.237097 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc"] Aug 01 00:44:32.920703 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:32.920664 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" event={"ID":"030c88c0-c802-40a3-8d3d-96428d108ff5","Type":"ContainerStarted","Data":"c7b6ac79ae977c83953c31b22f8a3f222f865d592b4940dd069eabc0eaba1bd7"} Aug 01 00:44:32.920913 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:32.920861 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:32.922067 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:32.922034 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" event={"ID":"52712577-618b-4f22-ba81-b3b515bbd2a3","Type":"ContainerStarted","Data":"04937afc1c6e8412db3fefe2bdd83000221e0daccd456b737dc2735679f926dd"} Aug 01 00:44:32.939912 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:32.939865 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" podStartSLOduration=2.075005188 podStartE2EDuration="4.93985185s" totalImagesPullingTime="2.864846662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:32.938396338 +0000 UTC m=+651.148024818" watchObservedRunningTime="2026-08-01 00:44:32.93985185 +0000 UTC m=+651.149480311" Aug 01 00:44:34.935794 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:34.935757 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" event={"ID":"52712577-618b-4f22-ba81-b3b515bbd2a3","Type":"ContainerStarted","Data":"87ec28bbc53ce0f39c0b9534dbab12ec34e9f2e15123765f86ee2b428436dc5f"} Aug 01 00:44:34.936234 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:34.935946 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" Aug 01 00:44:34.956739 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:34.956687 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" podStartSLOduration=2.937572928 podStartE2EDuration="4.956669384s" totalImagesPullingTime="2.019096456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:34.955114828 +0000 UTC m=+653.164743284" watchObservedRunningTime="2026-08-01 00:44:34.956669384 +0000 UTC m=+653.166297841" Aug 01 00:44:35.480314 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.480278 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt"] Aug 01 00:44:35.499491 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.499462 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:35.500126 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.500100 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt"] Aug 01 00:44:35.502389 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.502357 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8zn4g\"" Aug 01 00:44:35.654733 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.654696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" (UID: \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:35.654913 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.654744 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfnn\" (UniqueName: \"kubernetes.io/projected/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-kube-api-access-9sfnn\") pod \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" (UID: \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:35.755620 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.755532 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" (UID: \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:35.755620 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.755579 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfnn\" (UniqueName: \"kubernetes.io/projected/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-kube-api-access-9sfnn\") pod \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" (UID: \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:35.755942 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.755921 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" (UID: \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:35.763707 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.763675 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfnn\" (UniqueName: \"kubernetes.io/projected/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-kube-api-access-9sfnn\") pod \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" (UID: \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:35.811051 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.811006 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:35.951678 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:35.951646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt"] Aug 01 00:44:35.956065 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:35.956039 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9328ea5f_fc7a_406c_ab2d_c0c0d8dec888.slice/crio-2e0dd0cb4a603707f043a22cb9c7b07640c9aa6e4f48093f71057306bb696564 WatchSource:0}: Error finding container 2e0dd0cb4a603707f043a22cb9c7b07640c9aa6e4f48093f71057306bb696564: Status 404 returned error can't find the container with id 2e0dd0cb4a603707f043a22cb9c7b07640c9aa6e4f48093f71057306bb696564 Aug 01 00:44:36.948470 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:36.948431 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" event={"ID":"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888","Type":"ContainerStarted","Data":"2e0dd0cb4a603707f043a22cb9c7b07640c9aa6e4f48093f71057306bb696564"} Aug 01 00:44:38.081600 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.081563 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-9ztrl"] Aug 01 00:44:38.085187 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.085170 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" Aug 01 00:44:38.089074 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.089056 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8bcmt\"" Aug 01 00:44:38.100051 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.100030 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-9ztrl"] Aug 01 00:44:38.177769 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.177732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2ncq\" (UniqueName: \"kubernetes.io/projected/5bb3d5c0-5d81-4284-b683-db5971436d70-kube-api-access-t2ncq\") pod \"authorino-operator-68d88948d8-9ztrl\" (UID: \"5bb3d5c0-5d81-4284-b683-db5971436d70\") " pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" Aug 01 00:44:38.278680 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.278642 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2ncq\" (UniqueName: \"kubernetes.io/projected/5bb3d5c0-5d81-4284-b683-db5971436d70-kube-api-access-t2ncq\") pod \"authorino-operator-68d88948d8-9ztrl\" (UID: \"5bb3d5c0-5d81-4284-b683-db5971436d70\") " pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" Aug 01 00:44:38.292960 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.292927 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2ncq\" (UniqueName: \"kubernetes.io/projected/5bb3d5c0-5d81-4284-b683-db5971436d70-kube-api-access-t2ncq\") pod \"authorino-operator-68d88948d8-9ztrl\" (UID: \"5bb3d5c0-5d81-4284-b683-db5971436d70\") " pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" Aug 01 00:44:38.396485 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.396400 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" Aug 01 00:44:38.551807 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.551782 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-9ztrl"] Aug 01 00:44:38.553801 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:38.553769 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb3d5c0_5d81_4284_b683_db5971436d70.slice/crio-a2eecc40314a57c84286168482caa6dfded1ec52b3048552b2014be90a98f19c WatchSource:0}: Error finding container a2eecc40314a57c84286168482caa6dfded1ec52b3048552b2014be90a98f19c: Status 404 returned error can't find the container with id a2eecc40314a57c84286168482caa6dfded1ec52b3048552b2014be90a98f19c Aug 01 00:44:38.556146 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.556124 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:44:38.961107 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:38.961067 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" event={"ID":"5bb3d5c0-5d81-4284-b683-db5971436d70","Type":"ContainerStarted","Data":"a2eecc40314a57c84286168482caa6dfded1ec52b3048552b2014be90a98f19c"} Aug 01 00:44:40.977199 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:40.977155 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" event={"ID":"5bb3d5c0-5d81-4284-b683-db5971436d70","Type":"ContainerStarted","Data":"c3420bd3792746c5a2bde6bcc2e6d82e4e987a8aa647e687458ad1f60005547a"} Aug 01 00:44:40.977667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:40.977327 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" Aug 01 00:44:40.994669 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:40.994622 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" podStartSLOduration=0.936715533 podStartE2EDuration="2.994606134s" totalImagesPullingTime="2.057890601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:40.991571995 +0000 UTC m=+659.201200451" watchObservedRunningTime="2026-08-01 00:44:40.994606134 +0000 UTC m=+659.204234592" Aug 01 00:44:43.931904 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:43.931851 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:45.005835 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:45.005798 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" event={"ID":"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888","Type":"ContainerStarted","Data":"24f782ac8e1f830cd506ba420b215ac8c3a7dc93071e3d009f74eb6d32fed73e"} Aug 01 00:44:45.006270 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:45.005912 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:45.025276 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:45.025186 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" podStartSLOduration=1.082192896 podStartE2EDuration="10.025165773s" totalImagesPullingTime="8.942972877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:45.024440489 +0000 UTC m=+663.234068948" watchObservedRunningTime="2026-08-01 00:44:45.025165773 +0000 UTC m=+663.234794231" Aug 01 00:44:45.944327 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:45.944298 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zqsrc" Aug 01 00:44:47.607254 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.607204 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt"] Aug 01 00:44:47.610857 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.610838 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.613243 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.613201 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 01 00:44:47.613440 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.613425 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 01 00:44:47.613494 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.613433 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-677vz\"" Aug 01 00:44:47.618976 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.618948 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt"] Aug 01 00:44:47.664453 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.664416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.664453 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.664448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xsl\" (UniqueName: \"kubernetes.io/projected/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-kube-api-access-55xsl\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.664646 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.664544 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.765701 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.765665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.765906 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.765730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.765906 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.765752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xsl\" (UniqueName: \"kubernetes.io/projected/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-kube-api-access-55xsl\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.766418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.766394 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.768316 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.768295 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.773329 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.773311 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xsl\" (UniqueName: \"kubernetes.io/projected/4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e-kube-api-access-55xsl\") pod \"kuadrant-console-plugin-598f86468b-cplxt\" (UID: \"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:47.920916 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:47.920827 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" Aug 01 00:44:48.094315 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:48.094289 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt"] Aug 01 00:44:48.096329 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:48.096290 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cb76ea7_400a_4ab1_9979_53b6b9fe0a0e.slice/crio-15716783d2764f7aa6c7122afe835d33c420a7c45c6eb97cf6aa01e2cfc210a2 WatchSource:0}: Error finding container 15716783d2764f7aa6c7122afe835d33c420a7c45c6eb97cf6aa01e2cfc210a2: Status 404 returned error can't find the container with id 15716783d2764f7aa6c7122afe835d33c420a7c45c6eb97cf6aa01e2cfc210a2 Aug 01 00:44:49.030937 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:49.030851 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" event={"ID":"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e","Type":"ContainerStarted","Data":"15716783d2764f7aa6c7122afe835d33c420a7c45c6eb97cf6aa01e2cfc210a2"} Aug 01 00:44:51.984869 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:51.984837 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-9ztrl" Aug 01 00:44:56.013774 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:56.013744 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:57.825753 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.825704 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt"] Aug 01 00:44:57.826166 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.825902 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" containerName="manager" containerID="cri-o://24f782ac8e1f830cd506ba420b215ac8c3a7dc93071e3d009f74eb6d32fed73e" gracePeriod=2 Aug 01 00:44:57.833595 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.833564 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt"] Aug 01 00:44:57.849955 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.849928 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr"] Aug 01 00:44:57.850460 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.850439 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" containerName="manager" Aug 01 00:44:57.850548 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.850463 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" containerName="manager" Aug 01 00:44:57.850588 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.850577 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" containerName="manager" Aug 01 00:44:57.854236 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.854201 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:57.855945 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:44:57.855918 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" is forbidden: User \"system:node:ip-10-0-137-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-42.ec2.internal' and this object" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:57.867317 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.867291 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd"] Aug 01 00:44:57.867565 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.867520 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" podUID="030c88c0-c802-40a3-8d3d-96428d108ff5" containerName="manager" containerID="cri-o://c7b6ac79ae977c83953c31b22f8a3f222f865d592b4940dd069eabc0eaba1bd7" gracePeriod=2 Aug 01 00:44:57.870439 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.870419 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr"] Aug 01 00:44:57.883174 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:44:57.883140 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qxpsd\" is forbidden: User \"system:node:ip-10-0-137-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-42.ec2.internal' and this object" podUID="030c88c0-c802-40a3-8d3d-96428d108ff5" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:57.884878 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:44:57.884835 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" is forbidden: User \"system:node:ip-10-0-137-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-42.ec2.internal' and this object" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:57.887159 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.887139 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd"] Aug 01 00:44:57.958644 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.958617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2sf\" (UniqueName: \"kubernetes.io/projected/176d6541-f20d-49bd-8070-3d088c33f25d-kube-api-access-lc2sf\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhtjr\" (UID: \"176d6541-f20d-49bd-8070-3d088c33f25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:57.958767 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:57.958660 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176d6541-f20d-49bd-8070-3d088c33f25d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhtjr\" (UID: \"176d6541-f20d-49bd-8070-3d088c33f25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:58.059817 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.059780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176d6541-f20d-49bd-8070-3d088c33f25d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhtjr\" (UID: \"176d6541-f20d-49bd-8070-3d088c33f25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:58.059971 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.059904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2sf\" (UniqueName: \"kubernetes.io/projected/176d6541-f20d-49bd-8070-3d088c33f25d-kube-api-access-lc2sf\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhtjr\" (UID: \"176d6541-f20d-49bd-8070-3d088c33f25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:58.060136 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.060116 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176d6541-f20d-49bd-8070-3d088c33f25d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhtjr\" (UID: \"176d6541-f20d-49bd-8070-3d088c33f25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:58.068744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.068717 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2sf\" (UniqueName: \"kubernetes.io/projected/176d6541-f20d-49bd-8070-3d088c33f25d-kube-api-access-lc2sf\") pod \"kuadrant-operator-controller-manager-58fc56757c-rhtjr\" (UID: \"176d6541-f20d-49bd-8070-3d088c33f25d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:58.082835 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.082769 2568 generic.go:356] "Generic (PLEG): container finished" podID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" containerID="24f782ac8e1f830cd506ba420b215ac8c3a7dc93071e3d009f74eb6d32fed73e" exitCode=0 Aug 01 00:44:58.084588 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.084565 2568 generic.go:356] "Generic (PLEG): container finished" podID="030c88c0-c802-40a3-8d3d-96428d108ff5" containerID="c7b6ac79ae977c83953c31b22f8a3f222f865d592b4940dd069eabc0eaba1bd7" exitCode=0 Aug 01 00:44:58.110807 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.110781 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:58.112637 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:44:58.112614 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" is forbidden: User \"system:node:ip-10-0-137-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-42.ec2.internal' and this object" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:58.113668 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.113653 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:58.114026 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:44:58.114005 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qxpsd\" is forbidden: User \"system:node:ip-10-0-137-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-42.ec2.internal' and this object" podUID="030c88c0-c802-40a3-8d3d-96428d108ff5" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:58.115387 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:44:58.115367 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-cxblt\" is forbidden: User \"system:node:ip-10-0-137-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-42.ec2.internal' and this object" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:58.116661 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:44:58.116641 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-qxpsd\" is forbidden: User \"system:node:ip-10-0-137-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-42.ec2.internal' and this object" podUID="030c88c0-c802-40a3-8d3d-96428d108ff5" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:58.161157 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.161132 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sfnn\" (UniqueName: \"kubernetes.io/projected/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-kube-api-access-9sfnn\") pod \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\" (UID: \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\") " Aug 01 00:44:58.161279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.161191 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sbbf\" (UniqueName: \"kubernetes.io/projected/030c88c0-c802-40a3-8d3d-96428d108ff5-kube-api-access-2sbbf\") pod \"030c88c0-c802-40a3-8d3d-96428d108ff5\" (UID: \"030c88c0-c802-40a3-8d3d-96428d108ff5\") " Aug 01 00:44:58.161279 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.161229 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-extensions-socket-volume\") pod \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\" (UID: \"9328ea5f-fc7a-406c-ab2d-c0c0d8dec888\") " Aug 01 00:44:58.161931 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.161905 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" (UID: "9328ea5f-fc7a-406c-ab2d-c0c0d8dec888"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:44:58.163660 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.163638 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030c88c0-c802-40a3-8d3d-96428d108ff5-kube-api-access-2sbbf" (OuterVolumeSpecName: "kube-api-access-2sbbf") pod "030c88c0-c802-40a3-8d3d-96428d108ff5" (UID: "030c88c0-c802-40a3-8d3d-96428d108ff5"). InnerVolumeSpecName "kube-api-access-2sbbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:44:58.163805 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.163781 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-kube-api-access-9sfnn" (OuterVolumeSpecName: "kube-api-access-9sfnn") pod "9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" (UID: "9328ea5f-fc7a-406c-ab2d-c0c0d8dec888"). InnerVolumeSpecName "kube-api-access-9sfnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:44:58.231379 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.231348 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="030c88c0-c802-40a3-8d3d-96428d108ff5" path="/var/lib/kubelet/pods/030c88c0-c802-40a3-8d3d-96428d108ff5/volumes" Aug 01 00:44:58.231711 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.231699 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9328ea5f-fc7a-406c-ab2d-c0c0d8dec888" path="/var/lib/kubelet/pods/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888/volumes" Aug 01 00:44:58.252405 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.252384 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:58.262844 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.262825 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sbbf\" (UniqueName: \"kubernetes.io/projected/030c88c0-c802-40a3-8d3d-96428d108ff5-kube-api-access-2sbbf\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:58.262963 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.262850 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-extensions-socket-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:58.262963 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.262868 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sfnn\" (UniqueName: \"kubernetes.io/projected/9328ea5f-fc7a-406c-ab2d-c0c0d8dec888-kube-api-access-9sfnn\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:44:58.394794 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:58.394765 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr"] Aug 01 00:44:58.398229 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:44:58.398189 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176d6541_f20d_49bd_8070_3d088c33f25d.slice/crio-8b38809311f3f93c8d1941e9fa8c576236ba95485123e0892f7e6b8979588120 WatchSource:0}: Error finding container 8b38809311f3f93c8d1941e9fa8c576236ba95485123e0892f7e6b8979588120: Status 404 returned error can't find the container with id 8b38809311f3f93c8d1941e9fa8c576236ba95485123e0892f7e6b8979588120 Aug 01 00:44:59.093271 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:59.093243 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-cxblt" Aug 01 00:44:59.093735 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:59.093254 2568 scope.go:117] "RemoveContainer" containerID="24f782ac8e1f830cd506ba420b215ac8c3a7dc93071e3d009f74eb6d32fed73e" Aug 01 00:44:59.095531 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:59.095500 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" event={"ID":"176d6541-f20d-49bd-8070-3d088c33f25d","Type":"ContainerStarted","Data":"58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22"} Aug 01 00:44:59.095695 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:59.095562 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" event={"ID":"176d6541-f20d-49bd-8070-3d088c33f25d","Type":"ContainerStarted","Data":"8b38809311f3f93c8d1941e9fa8c576236ba95485123e0892f7e6b8979588120"} Aug 01 00:44:59.096240 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:59.096156 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:44:59.097568 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:59.097550 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-qxpsd" Aug 01 00:44:59.120860 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:44:59.120744 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" podStartSLOduration=2.120725287 podStartE2EDuration="2.120725287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:59.117704598 +0000 UTC m=+677.327333055" watchObservedRunningTime="2026-08-01 00:44:59.120725287 +0000 UTC m=+677.330353744" Aug 01 00:45:00.869969 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:00.869751 2568 scope.go:117] "RemoveContainer" containerID="c7b6ac79ae977c83953c31b22f8a3f222f865d592b4940dd069eabc0eaba1bd7" Aug 01 00:45:01.111489 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:01.111430 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" event={"ID":"4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e","Type":"ContainerStarted","Data":"6178d9747320dd2fb10f308c592da528394bc6efe4f54253cf9a6166ca4d2b96"} Aug 01 00:45:01.126986 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:01.126892 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-cplxt" podStartSLOduration=1.199232301 podStartE2EDuration="14.126875717s" totalImagesPullingTime="12.927643416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:01.125820934 +0000 UTC m=+679.335449389" watchObservedRunningTime="2026-08-01 00:45:01.126875717 +0000 UTC m=+679.336504173" Aug 01 00:45:11.115375 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:11.115343 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:45:14.743630 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:14.743600 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr"] Aug 01 00:45:14.744011 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:14.743784 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" podUID="176d6541-f20d-49bd-8070-3d088c33f25d" containerName="manager" containerID="cri-o://58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22" gracePeriod=10 Aug 01 00:45:14.996732 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:14.996669 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j"] Aug 01 00:45:14.997125 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:14.997107 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030c88c0-c802-40a3-8d3d-96428d108ff5" containerName="manager" Aug 01 00:45:14.997206 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:14.997127 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="030c88c0-c802-40a3-8d3d-96428d108ff5" containerName="manager" Aug 01 00:45:14.997289 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:14.997275 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="030c88c0-c802-40a3-8d3d-96428d108ff5" containerName="manager" Aug 01 00:45:15.001193 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.001174 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:15.010522 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.010502 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:45:15.014459 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.014441 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j"] Aug 01 00:45:15.125681 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.125650 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176d6541-f20d-49bd-8070-3d088c33f25d-extensions-socket-volume\") pod \"176d6541-f20d-49bd-8070-3d088c33f25d\" (UID: \"176d6541-f20d-49bd-8070-3d088c33f25d\") " Aug 01 00:45:15.125681 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.125686 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc2sf\" (UniqueName: \"kubernetes.io/projected/176d6541-f20d-49bd-8070-3d088c33f25d-kube-api-access-lc2sf\") pod \"176d6541-f20d-49bd-8070-3d088c33f25d\" (UID: \"176d6541-f20d-49bd-8070-3d088c33f25d\") " Aug 01 00:45:15.125932 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.125868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c6a63d-5000-4188-bfd7-8446ba3544dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-8gj6j\" (UID: \"07c6a63d-5000-4188-bfd7-8446ba3544dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:15.125997 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.125984 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwmcp\" (UniqueName: \"kubernetes.io/projected/07c6a63d-5000-4188-bfd7-8446ba3544dc-kube-api-access-gwmcp\") pod \"kuadrant-operator-controller-manager-68cf589589-8gj6j\" (UID: \"07c6a63d-5000-4188-bfd7-8446ba3544dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:15.126055 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.125987 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176d6541-f20d-49bd-8070-3d088c33f25d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "176d6541-f20d-49bd-8070-3d088c33f25d" (UID: "176d6541-f20d-49bd-8070-3d088c33f25d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:45:15.127971 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.127944 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176d6541-f20d-49bd-8070-3d088c33f25d-kube-api-access-lc2sf" (OuterVolumeSpecName: "kube-api-access-lc2sf") pod "176d6541-f20d-49bd-8070-3d088c33f25d" (UID: "176d6541-f20d-49bd-8070-3d088c33f25d"). InnerVolumeSpecName "kube-api-access-lc2sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:45:15.192629 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.192590 2568 generic.go:356] "Generic (PLEG): container finished" podID="176d6541-f20d-49bd-8070-3d088c33f25d" containerID="58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22" exitCode=0 Aug 01 00:45:15.192811 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.192639 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" Aug 01 00:45:15.192811 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.192686 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" event={"ID":"176d6541-f20d-49bd-8070-3d088c33f25d","Type":"ContainerDied","Data":"58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22"} Aug 01 00:45:15.192811 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.192737 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr" event={"ID":"176d6541-f20d-49bd-8070-3d088c33f25d","Type":"ContainerDied","Data":"8b38809311f3f93c8d1941e9fa8c576236ba95485123e0892f7e6b8979588120"} Aug 01 00:45:15.192811 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.192765 2568 scope.go:117] "RemoveContainer" containerID="58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22" Aug 01 00:45:15.202321 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.202304 2568 scope.go:117] "RemoveContainer" containerID="58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22" Aug 01 00:45:15.202564 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:45:15.202544 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22\": container with ID starting with 58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22 not found: ID does not exist" containerID="58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22" Aug 01 00:45:15.202619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.202574 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22"} err="failed to get container status \"58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22\": rpc error: code = NotFound desc = could not find container \"58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22\": container with ID starting with 58a07237108d1e6fd2aa96f5ee9635b5db9f214f95c2a69abb0ecab83d125c22 not found: ID does not exist" Aug 01 00:45:15.214137 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.214113 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr"] Aug 01 00:45:15.225978 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.221784 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-rhtjr"] Aug 01 00:45:15.226753 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.226730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c6a63d-5000-4188-bfd7-8446ba3544dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-8gj6j\" (UID: \"07c6a63d-5000-4188-bfd7-8446ba3544dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:15.226898 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.226884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwmcp\" (UniqueName: \"kubernetes.io/projected/07c6a63d-5000-4188-bfd7-8446ba3544dc-kube-api-access-gwmcp\") pod \"kuadrant-operator-controller-manager-68cf589589-8gj6j\" (UID: \"07c6a63d-5000-4188-bfd7-8446ba3544dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:15.226971 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.226937 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/176d6541-f20d-49bd-8070-3d088c33f25d-extensions-socket-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:45:15.226971 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.226954 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc2sf\" (UniqueName: \"kubernetes.io/projected/176d6541-f20d-49bd-8070-3d088c33f25d-kube-api-access-lc2sf\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:45:15.227076 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.227055 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c6a63d-5000-4188-bfd7-8446ba3544dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-8gj6j\" (UID: \"07c6a63d-5000-4188-bfd7-8446ba3544dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:15.235187 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.235165 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwmcp\" (UniqueName: \"kubernetes.io/projected/07c6a63d-5000-4188-bfd7-8446ba3544dc-kube-api-access-gwmcp\") pod \"kuadrant-operator-controller-manager-68cf589589-8gj6j\" (UID: \"07c6a63d-5000-4188-bfd7-8446ba3544dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:15.317037 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.316945 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:15.454991 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:15.454967 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j"] Aug 01 00:45:15.457072 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:45:15.457039 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c6a63d_5000_4188_bfd7_8446ba3544dc.slice/crio-6b43a6d9d6cd30bd4f5a895ac53e9693ba8095664f840a0cfd50e0108cc32ce1 WatchSource:0}: Error finding container 6b43a6d9d6cd30bd4f5a895ac53e9693ba8095664f840a0cfd50e0108cc32ce1: Status 404 returned error can't find the container with id 6b43a6d9d6cd30bd4f5a895ac53e9693ba8095664f840a0cfd50e0108cc32ce1 Aug 01 00:45:16.201030 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:16.201000 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" event={"ID":"07c6a63d-5000-4188-bfd7-8446ba3544dc","Type":"ContainerStarted","Data":"d1dfca845e59995e84df51cabf52340f8859b5e75bbc69285b6b59a1e15d90cb"} Aug 01 00:45:16.201030 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:16.201045 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" event={"ID":"07c6a63d-5000-4188-bfd7-8446ba3544dc","Type":"ContainerStarted","Data":"6b43a6d9d6cd30bd4f5a895ac53e9693ba8095664f840a0cfd50e0108cc32ce1"} Aug 01 00:45:16.201489 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:16.201081 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:16.223163 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:16.223118 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" podStartSLOduration=2.223103568 podStartE2EDuration="2.223103568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:16.221472671 +0000 UTC m=+694.431101131" watchObservedRunningTime="2026-08-01 00:45:16.223103568 +0000 UTC m=+694.432732025" Aug 01 00:45:16.231586 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:16.231557 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176d6541-f20d-49bd-8070-3d088c33f25d" path="/var/lib/kubelet/pods/176d6541-f20d-49bd-8070-3d088c33f25d/volumes" Aug 01 00:45:27.209069 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:27.209038 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:45:31.119260 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.119226 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn"] Aug 01 00:45:31.119856 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.119840 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176d6541-f20d-49bd-8070-3d088c33f25d" containerName="manager" Aug 01 00:45:31.119914 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.119859 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="176d6541-f20d-49bd-8070-3d088c33f25d" containerName="manager" Aug 01 00:45:31.119979 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.119969 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="176d6541-f20d-49bd-8070-3d088c33f25d" containerName="manager" Aug 01 00:45:31.123228 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.123194 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.125438 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.125413 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5slch\"" Aug 01 00:45:31.133527 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.133506 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn"] Aug 01 00:45:31.155295 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155262 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.155406 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.155406 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxs6\" (UniqueName: \"kubernetes.io/projected/f7871781-59fd-4cac-b923-1332ffb1785b-kube-api-access-txxs6\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.155406 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7871781-59fd-4cac-b923-1332ffb1785b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.155530 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.155530 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155480 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7871781-59fd-4cac-b923-1332ffb1785b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.155530 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7871781-59fd-4cac-b923-1332ffb1785b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.155676 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155544 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.155676 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.155577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256118 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256086 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256353 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256133 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7871781-59fd-4cac-b923-1332ffb1785b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256353 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7871781-59fd-4cac-b923-1332ffb1785b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256353 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256205 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256353 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256353 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256299 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256634 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256634 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxs6\" (UniqueName: \"kubernetes.io/projected/f7871781-59fd-4cac-b923-1332ffb1785b-kube-api-access-txxs6\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256634 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7871781-59fd-4cac-b923-1332ffb1785b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256793 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256703 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256793 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256772 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.256915 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.256889 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.257399 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.257377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7871781-59fd-4cac-b923-1332ffb1785b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.257444 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.257417 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.258830 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.258808 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7871781-59fd-4cac-b923-1332ffb1785b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.258928 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.258906 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7871781-59fd-4cac-b923-1332ffb1785b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.263352 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.263332 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7871781-59fd-4cac-b923-1332ffb1785b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.263487 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.263471 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxs6\" (UniqueName: \"kubernetes.io/projected/f7871781-59fd-4cac-b923-1332ffb1785b-kube-api-access-txxs6\") pod \"maas-default-gateway-openshift-default-78bb78ff78-z99dn\" (UID: \"f7871781-59fd-4cac-b923-1332ffb1785b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.437892 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.437869 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:31.576801 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.576777 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn"] Aug 01 00:45:31.578405 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:45:31.578375 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7871781_59fd_4cac_b923_1332ffb1785b.slice/crio-286194607e5dbc64dd853d8403cd63530f5e461879312b8c967fd052abbc0b13 WatchSource:0}: Error finding container 286194607e5dbc64dd853d8403cd63530f5e461879312b8c967fd052abbc0b13: Status 404 returned error can't find the container with id 286194607e5dbc64dd853d8403cd63530f5e461879312b8c967fd052abbc0b13 Aug 01 00:45:31.580738 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.580699 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:45:31.580834 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.580781 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:45:31.580834 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:31.580828 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:45:32.296833 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:32.296808 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" event={"ID":"f7871781-59fd-4cac-b923-1332ffb1785b","Type":"ContainerStarted","Data":"aed4f2c36b6bf56ada2d36a75700f5ebed8056f5273bc3fd18e350dcc31521cb"} Aug 01 00:45:32.297194 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:32.296857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" event={"ID":"f7871781-59fd-4cac-b923-1332ffb1785b","Type":"ContainerStarted","Data":"286194607e5dbc64dd853d8403cd63530f5e461879312b8c967fd052abbc0b13"} Aug 01 00:45:32.316810 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:32.316764 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" podStartSLOduration=1.316747084 podStartE2EDuration="1.316747084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:32.315825029 +0000 UTC m=+710.525453486" watchObservedRunningTime="2026-08-01 00:45:32.316747084 +0000 UTC m=+710.526375541" Aug 01 00:45:32.438814 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:32.438785 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:32.443558 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:32.443535 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:33.303250 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:33.303143 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:33.304065 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:33.304046 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:45:46.081434 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.081399 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:45:46.086700 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.086677 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:46.088764 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.088739 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 01 00:45:46.092694 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.092671 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:45:46.121350 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.121318 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:45:46.189752 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.189717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/181985b1-54d0-4e9f-9048-e8bd4c2c2e88-config-file\") pod \"limitador-limitador-85d9db4d5d-nhj5q\" (UID: \"181985b1-54d0-4e9f-9048-e8bd4c2c2e88\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:46.189932 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.189781 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ptt\" (UniqueName: \"kubernetes.io/projected/181985b1-54d0-4e9f-9048-e8bd4c2c2e88-kube-api-access-q2ptt\") pod \"limitador-limitador-85d9db4d5d-nhj5q\" (UID: \"181985b1-54d0-4e9f-9048-e8bd4c2c2e88\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:46.291355 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.291305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ptt\" (UniqueName: \"kubernetes.io/projected/181985b1-54d0-4e9f-9048-e8bd4c2c2e88-kube-api-access-q2ptt\") pod \"limitador-limitador-85d9db4d5d-nhj5q\" (UID: \"181985b1-54d0-4e9f-9048-e8bd4c2c2e88\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:46.291551 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.291477 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/181985b1-54d0-4e9f-9048-e8bd4c2c2e88-config-file\") pod \"limitador-limitador-85d9db4d5d-nhj5q\" (UID: \"181985b1-54d0-4e9f-9048-e8bd4c2c2e88\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:46.292076 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.292051 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/181985b1-54d0-4e9f-9048-e8bd4c2c2e88-config-file\") pod \"limitador-limitador-85d9db4d5d-nhj5q\" (UID: \"181985b1-54d0-4e9f-9048-e8bd4c2c2e88\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:46.298947 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.298926 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ptt\" (UniqueName: \"kubernetes.io/projected/181985b1-54d0-4e9f-9048-e8bd4c2c2e88-kube-api-access-q2ptt\") pod \"limitador-limitador-85d9db4d5d-nhj5q\" (UID: \"181985b1-54d0-4e9f-9048-e8bd4c2c2e88\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:46.398410 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.398300 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:46.537176 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.537151 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:45:46.539282 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:45:46.539256 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181985b1_54d0_4e9f_9048_e8bd4c2c2e88.slice/crio-1d8964aff0bb3d202cadcaf9e7d51a643e4d725843d7cac0042a4eb3c30547ee WatchSource:0}: Error finding container 1d8964aff0bb3d202cadcaf9e7d51a643e4d725843d7cac0042a4eb3c30547ee: Status 404 returned error can't find the container with id 1d8964aff0bb3d202cadcaf9e7d51a643e4d725843d7cac0042a4eb3c30547ee Aug 01 00:45:46.679598 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.679570 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vcnj"] Aug 01 00:45:46.684495 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.684470 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" Aug 01 00:45:46.686698 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.686675 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5lsdz\"" Aug 01 00:45:46.691469 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.691447 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vcnj"] Aug 01 00:45:46.796703 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.796662 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skcrc\" (UniqueName: \"kubernetes.io/projected/4bd052c6-c4b8-4d16-872f-61150b361bda-kube-api-access-skcrc\") pod \"authorino-c548b5d7d-8vcnj\" (UID: \"4bd052c6-c4b8-4d16-872f-61150b361bda\") " pod="kuadrant-system/authorino-c548b5d7d-8vcnj" Aug 01 00:45:46.898208 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.898166 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skcrc\" (UniqueName: \"kubernetes.io/projected/4bd052c6-c4b8-4d16-872f-61150b361bda-kube-api-access-skcrc\") pod \"authorino-c548b5d7d-8vcnj\" (UID: \"4bd052c6-c4b8-4d16-872f-61150b361bda\") " pod="kuadrant-system/authorino-c548b5d7d-8vcnj" Aug 01 00:45:46.906864 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.906836 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skcrc\" (UniqueName: \"kubernetes.io/projected/4bd052c6-c4b8-4d16-872f-61150b361bda-kube-api-access-skcrc\") pod \"authorino-c548b5d7d-8vcnj\" (UID: \"4bd052c6-c4b8-4d16-872f-61150b361bda\") " pod="kuadrant-system/authorino-c548b5d7d-8vcnj" Aug 01 00:45:46.996398 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:46.996307 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" Aug 01 00:45:47.131587 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:47.131550 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vcnj"] Aug 01 00:45:47.134455 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:45:47.134422 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bd052c6_c4b8_4d16_872f_61150b361bda.slice/crio-6dc451f1047a056adba6339967c55b51d4c94e4eb2526af3b22c8cd4a234d74a WatchSource:0}: Error finding container 6dc451f1047a056adba6339967c55b51d4c94e4eb2526af3b22c8cd4a234d74a: Status 404 returned error can't find the container with id 6dc451f1047a056adba6339967c55b51d4c94e4eb2526af3b22c8cd4a234d74a Aug 01 00:45:47.379380 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:47.379295 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" event={"ID":"181985b1-54d0-4e9f-9048-e8bd4c2c2e88","Type":"ContainerStarted","Data":"1d8964aff0bb3d202cadcaf9e7d51a643e4d725843d7cac0042a4eb3c30547ee"} Aug 01 00:45:47.380463 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:47.380434 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" event={"ID":"4bd052c6-c4b8-4d16-872f-61150b361bda","Type":"ContainerStarted","Data":"6dc451f1047a056adba6339967c55b51d4c94e4eb2526af3b22c8cd4a234d74a"} Aug 01 00:45:48.393455 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:48.393414 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" event={"ID":"181985b1-54d0-4e9f-9048-e8bd4c2c2e88","Type":"ContainerStarted","Data":"833c8021852a1f7695747d3a9a1fd27ed853083aa30d2c7e541dac03156c09cf"} Aug 01 00:45:48.393951 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:48.393640 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:45:48.410401 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:48.410344 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" podStartSLOduration=0.751584272 podStartE2EDuration="2.410322798s" totalImagesPullingTime="1.658738526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:48.408678167 +0000 UTC m=+726.618306625" watchObservedRunningTime="2026-08-01 00:45:48.410322798 +0000 UTC m=+726.619951267" Aug 01 00:45:49.400904 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:49.400801 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" event={"ID":"4bd052c6-c4b8-4d16-872f-61150b361bda","Type":"ContainerStarted","Data":"fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939"} Aug 01 00:45:49.419410 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:49.419361 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" podStartSLOduration=1.574482716 podStartE2EDuration="3.419346406s" totalImagesPullingTime="1.84486369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:45:49.413825015 +0000 UTC m=+727.623453471" watchObservedRunningTime="2026-08-01 00:45:49.419346406 +0000 UTC m=+727.628974862" Aug 01 00:45:59.401966 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:45:59.401933 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q" Aug 01 00:46:20.081278 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.081240 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vcnj"] Aug 01 00:46:20.081921 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.081471 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" podUID="4bd052c6-c4b8-4d16-872f-61150b361bda" containerName="authorino" containerID="cri-o://fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939" gracePeriod=30 Aug 01 00:46:20.326570 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.326544 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" Aug 01 00:46:20.412537 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.412463 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skcrc\" (UniqueName: \"kubernetes.io/projected/4bd052c6-c4b8-4d16-872f-61150b361bda-kube-api-access-skcrc\") pod \"4bd052c6-c4b8-4d16-872f-61150b361bda\" (UID: \"4bd052c6-c4b8-4d16-872f-61150b361bda\") " Aug 01 00:46:20.414416 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.414384 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd052c6-c4b8-4d16-872f-61150b361bda-kube-api-access-skcrc" (OuterVolumeSpecName: "kube-api-access-skcrc") pod "4bd052c6-c4b8-4d16-872f-61150b361bda" (UID: "4bd052c6-c4b8-4d16-872f-61150b361bda"). InnerVolumeSpecName "kube-api-access-skcrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:46:20.513842 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.513800 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skcrc\" (UniqueName: \"kubernetes.io/projected/4bd052c6-c4b8-4d16-872f-61150b361bda-kube-api-access-skcrc\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:46:20.583018 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.582986 2568 generic.go:356] "Generic (PLEG): container finished" podID="4bd052c6-c4b8-4d16-872f-61150b361bda" containerID="fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939" exitCode=0 Aug 01 00:46:20.583174 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.583032 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" Aug 01 00:46:20.583174 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.583050 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" event={"ID":"4bd052c6-c4b8-4d16-872f-61150b361bda","Type":"ContainerDied","Data":"fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939"} Aug 01 00:46:20.583174 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.583083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8vcnj" event={"ID":"4bd052c6-c4b8-4d16-872f-61150b361bda","Type":"ContainerDied","Data":"6dc451f1047a056adba6339967c55b51d4c94e4eb2526af3b22c8cd4a234d74a"} Aug 01 00:46:20.583174 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.583101 2568 scope.go:117] "RemoveContainer" containerID="fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939" Aug 01 00:46:20.593084 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.593059 2568 scope.go:117] "RemoveContainer" containerID="fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939" Aug 01 00:46:20.593391 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:46:20.593368 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939\": container with ID starting with fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939 not found: ID does not exist" containerID="fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939" Aug 01 00:46:20.593476 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.593403 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939"} err="failed to get container status \"fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939\": rpc error: code = NotFound desc = could not find container \"fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939\": container with ID starting with fc066989f560cd2ead18345389f395bbf2db030f198800baae4e86c141412939 not found: ID does not exist" Aug 01 00:46:20.602858 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.602836 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vcnj"] Aug 01 00:46:20.605101 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:20.605082 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vcnj"] Aug 01 00:46:22.231205 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:22.231171 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bd052c6-c4b8-4d16-872f-61150b361bda" path="/var/lib/kubelet/pods/4bd052c6-c4b8-4d16-872f-61150b361bda/volumes" Aug 01 00:46:24.792856 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.792823 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d79d969f-hr7xf"] Aug 01 00:46:24.793334 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.793318 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bd052c6-c4b8-4d16-872f-61150b361bda" containerName="authorino" Aug 01 00:46:24.793334 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.793331 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bd052c6-c4b8-4d16-872f-61150b361bda" containerName="authorino" Aug 01 00:46:24.793458 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.793403 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bd052c6-c4b8-4d16-872f-61150b361bda" containerName="authorino" Aug 01 00:46:24.799615 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.799594 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:24.801924 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.801900 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 01 00:46:24.802048 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.801980 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ldrkx\"" Aug 01 00:46:24.802548 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.802529 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 01 00:46:24.807054 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.807031 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79d969f-hr7xf"] Aug 01 00:46:24.858425 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.858379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d972e615-1dd2-4d96-bf92-82b25fabab4a-webhook-cert\") pod \"maas-controller-d79d969f-hr7xf\" (UID: \"d972e615-1dd2-4d96-bf92-82b25fabab4a\") " pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:24.858582 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.858499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdtlp\" (UniqueName: \"kubernetes.io/projected/d972e615-1dd2-4d96-bf92-82b25fabab4a-kube-api-access-qdtlp\") pod \"maas-controller-d79d969f-hr7xf\" (UID: \"d972e615-1dd2-4d96-bf92-82b25fabab4a\") " pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:24.959633 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.959594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d972e615-1dd2-4d96-bf92-82b25fabab4a-webhook-cert\") pod \"maas-controller-d79d969f-hr7xf\" (UID: \"d972e615-1dd2-4d96-bf92-82b25fabab4a\") " pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:24.959803 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.959657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtlp\" (UniqueName: \"kubernetes.io/projected/d972e615-1dd2-4d96-bf92-82b25fabab4a-kube-api-access-qdtlp\") pod \"maas-controller-d79d969f-hr7xf\" (UID: \"d972e615-1dd2-4d96-bf92-82b25fabab4a\") " pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:24.962401 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.962377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d972e615-1dd2-4d96-bf92-82b25fabab4a-webhook-cert\") pod \"maas-controller-d79d969f-hr7xf\" (UID: \"d972e615-1dd2-4d96-bf92-82b25fabab4a\") " pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:24.971465 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:24.971435 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtlp\" (UniqueName: \"kubernetes.io/projected/d972e615-1dd2-4d96-bf92-82b25fabab4a-kube-api-access-qdtlp\") pod \"maas-controller-d79d969f-hr7xf\" (UID: \"d972e615-1dd2-4d96-bf92-82b25fabab4a\") " pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:25.113496 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:25.113400 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:25.254607 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:25.254582 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79d969f-hr7xf"] Aug 01 00:46:25.256490 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:46:25.256463 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd972e615_1dd2_4d96_bf92_82b25fabab4a.slice/crio-a21be29680280bdfb4b6bd1108c6db485d19a369a4227a92ceb9c57b9d887651 WatchSource:0}: Error finding container a21be29680280bdfb4b6bd1108c6db485d19a369a4227a92ceb9c57b9d887651: Status 404 returned error can't find the container with id a21be29680280bdfb4b6bd1108c6db485d19a369a4227a92ceb9c57b9d887651 Aug 01 00:46:25.615509 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:25.615473 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79d969f-hr7xf" event={"ID":"d972e615-1dd2-4d96-bf92-82b25fabab4a","Type":"ContainerStarted","Data":"a21be29680280bdfb4b6bd1108c6db485d19a369a4227a92ceb9c57b9d887651"} Aug 01 00:46:27.630909 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:27.630872 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79d969f-hr7xf" event={"ID":"d972e615-1dd2-4d96-bf92-82b25fabab4a","Type":"ContainerStarted","Data":"f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c"} Aug 01 00:46:27.631406 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:27.630935 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:27.667446 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:27.667378 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d79d969f-hr7xf" podStartSLOduration=1.962643133 podStartE2EDuration="3.66735998s" totalImagesPullingTime="1.704716847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:46:27.665295744 +0000 UTC m=+765.874924200" watchObservedRunningTime="2026-08-01 00:46:27.66735998 +0000 UTC m=+765.876988436" Aug 01 00:46:38.128874 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.128836 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d79d969f-hr7xf"] Aug 01 00:46:38.129316 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.129050 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d79d969f-hr7xf" podUID="d972e615-1dd2-4d96-bf92-82b25fabab4a" containerName="manager" containerID="cri-o://f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c" gracePeriod=30 Aug 01 00:46:38.130856 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.130833 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:38.378013 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.377991 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:38.488601 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.488558 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdtlp\" (UniqueName: \"kubernetes.io/projected/d972e615-1dd2-4d96-bf92-82b25fabab4a-kube-api-access-qdtlp\") pod \"d972e615-1dd2-4d96-bf92-82b25fabab4a\" (UID: \"d972e615-1dd2-4d96-bf92-82b25fabab4a\") " Aug 01 00:46:38.488770 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.488625 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d972e615-1dd2-4d96-bf92-82b25fabab4a-webhook-cert\") pod \"d972e615-1dd2-4d96-bf92-82b25fabab4a\" (UID: \"d972e615-1dd2-4d96-bf92-82b25fabab4a\") " Aug 01 00:46:38.491032 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.491005 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d972e615-1dd2-4d96-bf92-82b25fabab4a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d972e615-1dd2-4d96-bf92-82b25fabab4a" (UID: "d972e615-1dd2-4d96-bf92-82b25fabab4a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:46:38.491143 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.491043 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d972e615-1dd2-4d96-bf92-82b25fabab4a-kube-api-access-qdtlp" (OuterVolumeSpecName: "kube-api-access-qdtlp") pod "d972e615-1dd2-4d96-bf92-82b25fabab4a" (UID: "d972e615-1dd2-4d96-bf92-82b25fabab4a"). InnerVolumeSpecName "kube-api-access-qdtlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:46:38.590063 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.590027 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdtlp\" (UniqueName: \"kubernetes.io/projected/d972e615-1dd2-4d96-bf92-82b25fabab4a-kube-api-access-qdtlp\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:46:38.590063 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.590055 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d972e615-1dd2-4d96-bf92-82b25fabab4a-webhook-cert\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:46:38.699068 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.699031 2568 generic.go:356] "Generic (PLEG): container finished" podID="d972e615-1dd2-4d96-bf92-82b25fabab4a" containerID="f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c" exitCode=0 Aug 01 00:46:38.699294 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.699093 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79d969f-hr7xf" Aug 01 00:46:38.699294 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.699115 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79d969f-hr7xf" event={"ID":"d972e615-1dd2-4d96-bf92-82b25fabab4a","Type":"ContainerDied","Data":"f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c"} Aug 01 00:46:38.699294 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.699190 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79d969f-hr7xf" event={"ID":"d972e615-1dd2-4d96-bf92-82b25fabab4a","Type":"ContainerDied","Data":"a21be29680280bdfb4b6bd1108c6db485d19a369a4227a92ceb9c57b9d887651"} Aug 01 00:46:38.699294 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.699235 2568 scope.go:117] "RemoveContainer" containerID="f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c" Aug 01 00:46:38.708920 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.708898 2568 scope.go:117] "RemoveContainer" containerID="f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c" Aug 01 00:46:38.709161 ip-10-0-137-42 kubenswrapper[2568]: E0801 00:46:38.709145 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c\": container with ID starting with f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c not found: ID does not exist" containerID="f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c" Aug 01 00:46:38.709245 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.709169 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c"} err="failed to get container status \"f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c\": rpc error: code = NotFound desc = could not find container \"f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c\": container with ID starting with f880927005f02193c5444fe9343466ba589d386a684e794dfb1580af920a1e4c not found: ID does not exist" Aug 01 00:46:38.721311 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.721265 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d79d969f-hr7xf"] Aug 01 00:46:38.724904 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:38.724883 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d79d969f-hr7xf"] Aug 01 00:46:40.231056 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:40.231024 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d972e615-1dd2-4d96-bf92-82b25fabab4a" path="/var/lib/kubelet/pods/d972e615-1dd2-4d96-bf92-82b25fabab4a/volumes" Aug 01 00:46:58.524163 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.524121 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:46:58.538383 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.538352 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9wqfm"] Aug 01 00:46:58.539031 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.539001 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d972e615-1dd2-4d96-bf92-82b25fabab4a" containerName="manager" Aug 01 00:46:58.539031 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.539031 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d972e615-1dd2-4d96-bf92-82b25fabab4a" containerName="manager" Aug 01 00:46:58.539194 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.539151 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d972e615-1dd2-4d96-bf92-82b25fabab4a" containerName="manager" Aug 01 00:46:58.545847 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.545827 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:46:58.548494 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.548471 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-prjrx\"" Aug 01 00:46:58.548590 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.548575 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 01 00:46:58.554394 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.554361 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d26da9-9d38-4531-8ad6-cff1582092b8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9wqfm\" (UID: \"21d26da9-9d38-4531-8ad6-cff1582092b8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:46:58.554527 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.554419 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshwk\" (UniqueName: \"kubernetes.io/projected/21d26da9-9d38-4531-8ad6-cff1582092b8-kube-api-access-tshwk\") pod \"payload-processing-5cc6cd444b-9wqfm\" (UID: \"21d26da9-9d38-4531-8ad6-cff1582092b8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:46:58.554812 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.554789 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9wqfm"] Aug 01 00:46:58.655663 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.655627 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d26da9-9d38-4531-8ad6-cff1582092b8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9wqfm\" (UID: \"21d26da9-9d38-4531-8ad6-cff1582092b8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:46:58.655849 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.655670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshwk\" (UniqueName: \"kubernetes.io/projected/21d26da9-9d38-4531-8ad6-cff1582092b8-kube-api-access-tshwk\") pod \"payload-processing-5cc6cd444b-9wqfm\" (UID: \"21d26da9-9d38-4531-8ad6-cff1582092b8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:46:58.656357 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.656335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d26da9-9d38-4531-8ad6-cff1582092b8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-9wqfm\" (UID: \"21d26da9-9d38-4531-8ad6-cff1582092b8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:46:58.663095 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.663070 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshwk\" (UniqueName: \"kubernetes.io/projected/21d26da9-9d38-4531-8ad6-cff1582092b8-kube-api-access-tshwk\") pod \"payload-processing-5cc6cd444b-9wqfm\" (UID: \"21d26da9-9d38-4531-8ad6-cff1582092b8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:46:58.857644 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.857556 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:46:58.997707 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:58.997683 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-9wqfm"] Aug 01 00:46:58.999366 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:46:58.999339 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d26da9_9d38_4531_8ad6_cff1582092b8.slice/crio-b50dbb5934b11c18068112dcfaacc7e4b92e95e3d77196a8788a29dbf99ece5d WatchSource:0}: Error finding container b50dbb5934b11c18068112dcfaacc7e4b92e95e3d77196a8788a29dbf99ece5d: Status 404 returned error can't find the container with id b50dbb5934b11c18068112dcfaacc7e4b92e95e3d77196a8788a29dbf99ece5d Aug 01 00:46:59.823180 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:46:59.823137 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" event={"ID":"21d26da9-9d38-4531-8ad6-cff1582092b8","Type":"ContainerStarted","Data":"b50dbb5934b11c18068112dcfaacc7e4b92e95e3d77196a8788a29dbf99ece5d"} Aug 01 00:47:01.837034 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:01.836991 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" event={"ID":"21d26da9-9d38-4531-8ad6-cff1582092b8","Type":"ContainerStarted","Data":"c88cfd684f4ff64b379205f13cc0d58748a8b30bef6d9a556f8d2bc9a1bd9fbf"} Aug 01 00:47:01.837485 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:01.837124 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:47:01.854708 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:01.854640 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" podStartSLOduration=1.978889772 podStartE2EDuration="3.854617121s" totalImagesPullingTime="1.875727349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:47:01.851421218 +0000 UTC m=+800.061049676" watchObservedRunningTime="2026-08-01 00:47:01.854617121 +0000 UTC m=+800.064245579" Aug 01 00:47:08.889175 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:08.889138 2568 generic.go:356] "Generic (PLEG): container finished" podID="f7871781-59fd-4cac-b923-1332ffb1785b" containerID="aed4f2c36b6bf56ada2d36a75700f5ebed8056f5273bc3fd18e350dcc31521cb" exitCode=0 Aug 01 00:47:08.889612 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:08.889225 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" event={"ID":"f7871781-59fd-4cac-b923-1332ffb1785b","Type":"ContainerDied","Data":"aed4f2c36b6bf56ada2d36a75700f5ebed8056f5273bc3fd18e350dcc31521cb"} Aug 01 00:47:08.889666 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:08.889611 2568 scope.go:117] "RemoveContainer" containerID="aed4f2c36b6bf56ada2d36a75700f5ebed8056f5273bc3fd18e350dcc31521cb" Aug 01 00:47:08.890617 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:08.890593 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:47:08.890667 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:08.890657 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:47:08.890711 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:08.890687 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 00:47:09.896937 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:09.896898 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" event={"ID":"f7871781-59fd-4cac-b923-1332ffb1785b","Type":"ContainerStarted","Data":"413a648d03d54aef7e4933e579d69acbbbb8815fe850f2967d1ac5ebc06ad1ca"} Aug 01 00:47:10.438825 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:10.438796 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:47:10.439425 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:10.439111 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-z99dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Aug 01 00:47:10.439425 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:10.439167 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" podUID="f7871781-59fd-4cac-b923-1332ffb1785b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Aug 01 00:47:11.438746 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:11.438702 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-z99dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Aug 01 00:47:11.439263 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:11.438777 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" podUID="f7871781-59fd-4cac-b923-1332ffb1785b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Aug 01 00:47:12.453630 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:12.453589 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-z99dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Aug 01 00:47:12.454116 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:12.453673 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" podUID="f7871781-59fd-4cac-b923-1332ffb1785b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Aug 01 00:47:12.848703 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:12.848601 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-9wqfm" Aug 01 00:47:13.438520 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:13.438483 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-z99dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Aug 01 00:47:13.438721 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:13.438560 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" podUID="f7871781-59fd-4cac-b923-1332ffb1785b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Aug 01 00:47:14.438462 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:14.438407 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-z99dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Aug 01 00:47:14.438967 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:14.438489 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" podUID="f7871781-59fd-4cac-b923-1332ffb1785b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Aug 01 00:47:15.443310 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:15.443277 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:47:15.443754 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:15.443733 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:47:15.444471 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:15.444452 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-z99dn" Aug 01 00:47:26.583734 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.583704 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz"] Aug 01 00:47:26.587694 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.587675 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.590539 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.590513 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h6wmw\"" Aug 01 00:47:26.590662 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.590514 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 01 00:47:26.590662 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.590519 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 01 00:47:26.590662 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.590570 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 01 00:47:26.596951 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.596931 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz"] Aug 01 00:47:26.721248 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.721193 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gltgr\" (UniqueName: \"kubernetes.io/projected/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-kube-api-access-gltgr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.721418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.721298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.721418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.721320 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.721418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.721350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.721418 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.721382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.721606 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.721444 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822287 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822472 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822347 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gltgr\" (UniqueName: \"kubernetes.io/projected/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-kube-api-access-gltgr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822472 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822399 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822472 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822472 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822461 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822685 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822829 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822808 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.822892 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.822832 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.824685 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.824665 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.825037 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.825017 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.829051 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.829030 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gltgr\" (UniqueName: \"kubernetes.io/projected/f8b977f4-b3e4-41ff-bf88-5a69fb1128f9-kube-api-access-gltgr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz\" (UID: \"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:26.899850 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:26.899771 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:27.041697 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:27.041659 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz"] Aug 01 00:47:27.043621 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:47:27.043592 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b977f4_b3e4_41ff_bf88_5a69fb1128f9.slice/crio-99a171de361a6a7156e6d8dca79135f767b8e13718b4c03a0ec5223ee4f6a18b WatchSource:0}: Error finding container 99a171de361a6a7156e6d8dca79135f767b8e13718b4c03a0ec5223ee4f6a18b: Status 404 returned error can't find the container with id 99a171de361a6a7156e6d8dca79135f767b8e13718b4c03a0ec5223ee4f6a18b Aug 01 00:47:27.090619 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:27.090588 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:47:28.010942 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:28.010901 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" event={"ID":"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9","Type":"ContainerStarted","Data":"99a171de361a6a7156e6d8dca79135f767b8e13718b4c03a0ec5223ee4f6a18b"} Aug 01 00:47:29.018342 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:29.018305 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" event={"ID":"f8b977f4-b3e4-41ff-bf88-5a69fb1128f9","Type":"ContainerStarted","Data":"9ec23a0049bcfcee8503fbfeaa76bad76c50d2c9c76cc3df7edf39570f14d0aa"} Aug 01 00:47:29.035257 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:29.035188 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" podStartSLOduration=1.949395339 podStartE2EDuration="3.03516998s" totalImagesPullingTime="1.085774641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:47:29.032593645 +0000 UTC m=+827.242222103" watchObservedRunningTime="2026-08-01 00:47:29.03516998 +0000 UTC m=+827.244798437" Aug 01 00:47:36.900768 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:36.900721 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:36.901298 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:36.900790 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:36.913501 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:36.913477 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:47:37.077323 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:47:37.077291 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz" Aug 01 00:48:11.595422 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:48:11.595388 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:48:19.494496 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:48:19.494461 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:48:33.896113 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:48:33.896029 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:48:44.997116 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:48:44.997074 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:49:08.200158 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.200121 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd"] Aug 01 00:49:08.204305 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.204274 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.206644 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.206624 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 01 00:49:08.214029 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.214001 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd"] Aug 01 00:49:08.282980 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.282935 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.283176 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.283008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.283176 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.283087 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.283176 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.283153 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.384581 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.384542 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.384761 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.384587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d35dc51-5b10-48d7-98e0-f8f148122863-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.384761 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.384638 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.384761 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.384665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.384761 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.384737 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.384965 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.384783 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj6br\" (UniqueName: \"kubernetes.io/projected/5d35dc51-5b10-48d7-98e0-f8f148122863-kube-api-access-pj6br\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.385030 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.384965 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.385030 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.385003 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.385201 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.385151 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.387016 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.386996 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d35dc51-5b10-48d7-98e0-f8f148122863-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.486125 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.486039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6br\" (UniqueName: \"kubernetes.io/projected/5d35dc51-5b10-48d7-98e0-f8f148122863-kube-api-access-pj6br\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.486125 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.486117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d35dc51-5b10-48d7-98e0-f8f148122863-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.488757 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.488736 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d35dc51-5b10-48d7-98e0-f8f148122863-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.493850 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.493831 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6br\" (UniqueName: \"kubernetes.io/projected/5d35dc51-5b10-48d7-98e0-f8f148122863-kube-api-access-pj6br\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd\" (UID: \"5d35dc51-5b10-48d7-98e0-f8f148122863\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.514806 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.514776 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:08.660436 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:08.660408 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd"] Aug 01 00:49:08.662558 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:49:08.662528 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d35dc51_5b10_48d7_98e0_f8f148122863.slice/crio-0b0b5cbb96e52c3f14bf529565f8909af838b1127cfb9c8dbb4539e46babe8c8 WatchSource:0}: Error finding container 0b0b5cbb96e52c3f14bf529565f8909af838b1127cfb9c8dbb4539e46babe8c8: Status 404 returned error can't find the container with id 0b0b5cbb96e52c3f14bf529565f8909af838b1127cfb9c8dbb4539e46babe8c8 Aug 01 00:49:09.599615 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:09.599574 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" event={"ID":"5d35dc51-5b10-48d7-98e0-f8f148122863","Type":"ContainerStarted","Data":"c4576beaeb0fd2ccf5f355246397090b4d4abb893136c305c9583f14b201b334"} Aug 01 00:49:09.599615 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:09.599617 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" event={"ID":"5d35dc51-5b10-48d7-98e0-f8f148122863","Type":"ContainerStarted","Data":"0b0b5cbb96e52c3f14bf529565f8909af838b1127cfb9c8dbb4539e46babe8c8"} Aug 01 00:49:09.618808 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:09.618757 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" podStartSLOduration=1.416842059 podStartE2EDuration="1.618742763s" totalImagesPullingTime="201.900704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:49:09.616492161 +0000 UTC m=+927.826120618" watchObservedRunningTime="2026-08-01 00:49:09.618742763 +0000 UTC m=+927.828371222" Aug 01 00:49:10.001357 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:10.001322 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:49:18.515777 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:18.515737 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:18.516294 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:18.515790 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:18.528698 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:18.528673 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:49:18.666607 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:49:18.666574 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd" Aug 01 00:50:42.093835 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:50:42.093796 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:50:51.090016 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:50:51.089980 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:51:02.896792 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:51:02.896759 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:52:39.799270 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:52:39.799240 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:52:47.786992 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:52:47.786959 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:53:49.607198 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:53:49.607165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:53:56.896629 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:53:56.893366 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:54:12.232112 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:54:12.232079 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:54:18.199822 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:54:18.199784 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:54:27.796291 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:54:27.796252 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:54:34.603663 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:54:34.603576 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:54:48.294154 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:54:48.294114 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:54:53.601914 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:54:53.601877 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:54:58.997623 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:54:58.997584 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:55:20.300262 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:55:20.300223 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:55:31.187920 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:55:31.187883 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:55:53.701380 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:55:53.701332 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:56:04.200262 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:56:04.200157 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:56:10.698282 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:56:10.698245 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:56:17.205733 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:56:17.205691 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:56:24.993055 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:56:24.993020 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:56:30.798444 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:56:30.798403 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:56:35.902896 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:56:35.902854 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:56:43.197470 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:56:43.197426 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:56:54.807858 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:56:54.807823 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:57:02.396260 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:57:02.396224 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:57:13.095224 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:57:13.095188 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:57:18.300526 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:57:18.300483 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:57:24.805191 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:57:24.805154 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:57:33.409346 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:57:33.409263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:57:38.594752 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:57:38.594714 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:57:50.195620 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:57:50.195584 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 00:58:44.909928 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:44.909895 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j"] Aug 01 00:58:44.910439 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:44.910094 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" podUID="07c6a63d-5000-4188-bfd7-8446ba3544dc" containerName="manager" containerID="cri-o://d1dfca845e59995e84df51cabf52340f8859b5e75bbc69285b6b59a1e15d90cb" gracePeriod=10 Aug 01 00:58:45.065812 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.065778 2568 generic.go:356] "Generic (PLEG): container finished" podID="07c6a63d-5000-4188-bfd7-8446ba3544dc" containerID="d1dfca845e59995e84df51cabf52340f8859b5e75bbc69285b6b59a1e15d90cb" exitCode=0 Aug 01 00:58:45.065989 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.065855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" event={"ID":"07c6a63d-5000-4188-bfd7-8446ba3544dc","Type":"ContainerDied","Data":"d1dfca845e59995e84df51cabf52340f8859b5e75bbc69285b6b59a1e15d90cb"} Aug 01 00:58:45.169845 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.169778 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:58:45.315938 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.315900 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c6a63d-5000-4188-bfd7-8446ba3544dc-extensions-socket-volume\") pod \"07c6a63d-5000-4188-bfd7-8446ba3544dc\" (UID: \"07c6a63d-5000-4188-bfd7-8446ba3544dc\") " Aug 01 00:58:45.316146 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.315979 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwmcp\" (UniqueName: \"kubernetes.io/projected/07c6a63d-5000-4188-bfd7-8446ba3544dc-kube-api-access-gwmcp\") pod \"07c6a63d-5000-4188-bfd7-8446ba3544dc\" (UID: \"07c6a63d-5000-4188-bfd7-8446ba3544dc\") " Aug 01 00:58:45.316336 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.316312 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c6a63d-5000-4188-bfd7-8446ba3544dc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "07c6a63d-5000-4188-bfd7-8446ba3544dc" (UID: "07c6a63d-5000-4188-bfd7-8446ba3544dc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:58:45.318226 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.318190 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c6a63d-5000-4188-bfd7-8446ba3544dc-kube-api-access-gwmcp" (OuterVolumeSpecName: "kube-api-access-gwmcp") pod "07c6a63d-5000-4188-bfd7-8446ba3544dc" (UID: "07c6a63d-5000-4188-bfd7-8446ba3544dc"). InnerVolumeSpecName "kube-api-access-gwmcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:58:45.417542 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.417505 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwmcp\" (UniqueName: \"kubernetes.io/projected/07c6a63d-5000-4188-bfd7-8446ba3544dc-kube-api-access-gwmcp\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:58:45.417542 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:45.417537 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c6a63d-5000-4188-bfd7-8446ba3544dc-extensions-socket-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 00:58:46.072744 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:46.072716 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" Aug 01 00:58:46.073242 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:46.072759 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j" event={"ID":"07c6a63d-5000-4188-bfd7-8446ba3544dc","Type":"ContainerDied","Data":"6b43a6d9d6cd30bd4f5a895ac53e9693ba8095664f840a0cfd50e0108cc32ce1"} Aug 01 00:58:46.073242 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:46.072818 2568 scope.go:117] "RemoveContainer" containerID="d1dfca845e59995e84df51cabf52340f8859b5e75bbc69285b6b59a1e15d90cb" Aug 01 00:58:46.102014 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:46.101982 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j"] Aug 01 00:58:46.106339 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:46.106316 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8gj6j"] Aug 01 00:58:46.231669 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:58:46.231639 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c6a63d-5000-4188-bfd7-8446ba3544dc" path="/var/lib/kubelet/pods/07c6a63d-5000-4188-bfd7-8446ba3544dc/volumes" Aug 01 00:59:45.634080 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.634043 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm"] Aug 01 00:59:45.634544 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.634530 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c6a63d-5000-4188-bfd7-8446ba3544dc" containerName="manager" Aug 01 00:59:45.634544 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.634546 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c6a63d-5000-4188-bfd7-8446ba3544dc" containerName="manager" Aug 01 00:59:45.634652 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.634642 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c6a63d-5000-4188-bfd7-8446ba3544dc" containerName="manager" Aug 01 00:59:45.638831 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.638815 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:45.641041 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.641020 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8zn4g\"" Aug 01 00:59:45.648340 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.648316 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm"] Aug 01 00:59:45.693525 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.693482 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l2xd\" (UniqueName: \"kubernetes.io/projected/ba879383-aa83-42bd-887f-8ff3214fbc6a-kube-api-access-2l2xd\") pod \"kuadrant-operator-controller-manager-68cf589589-gw6qm\" (UID: \"ba879383-aa83-42bd-887f-8ff3214fbc6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:45.693707 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.693540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba879383-aa83-42bd-887f-8ff3214fbc6a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-gw6qm\" (UID: \"ba879383-aa83-42bd-887f-8ff3214fbc6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:45.794749 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.794700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l2xd\" (UniqueName: \"kubernetes.io/projected/ba879383-aa83-42bd-887f-8ff3214fbc6a-kube-api-access-2l2xd\") pod \"kuadrant-operator-controller-manager-68cf589589-gw6qm\" (UID: \"ba879383-aa83-42bd-887f-8ff3214fbc6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:45.794749 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.794762 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba879383-aa83-42bd-887f-8ff3214fbc6a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-gw6qm\" (UID: \"ba879383-aa83-42bd-887f-8ff3214fbc6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:45.795114 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.795093 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba879383-aa83-42bd-887f-8ff3214fbc6a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-gw6qm\" (UID: \"ba879383-aa83-42bd-887f-8ff3214fbc6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:45.804835 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.804802 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l2xd\" (UniqueName: \"kubernetes.io/projected/ba879383-aa83-42bd-887f-8ff3214fbc6a-kube-api-access-2l2xd\") pod \"kuadrant-operator-controller-manager-68cf589589-gw6qm\" (UID: \"ba879383-aa83-42bd-887f-8ff3214fbc6a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:45.950070 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:45.950036 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:46.107094 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:46.107060 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm"] Aug 01 00:59:46.109137 ip-10-0-137-42 kubenswrapper[2568]: W0801 00:59:46.109104 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba879383_aa83_42bd_887f_8ff3214fbc6a.slice/crio-a2f1bf7106bc2a1ab8e6ce828bb489028695fe9d5188972ceadb8210881d4d90 WatchSource:0}: Error finding container a2f1bf7106bc2a1ab8e6ce828bb489028695fe9d5188972ceadb8210881d4d90: Status 404 returned error can't find the container with id a2f1bf7106bc2a1ab8e6ce828bb489028695fe9d5188972ceadb8210881d4d90 Aug 01 00:59:46.422865 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:46.422768 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" event={"ID":"ba879383-aa83-42bd-887f-8ff3214fbc6a","Type":"ContainerStarted","Data":"7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1"} Aug 01 00:59:46.422865 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:46.422829 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" event={"ID":"ba879383-aa83-42bd-887f-8ff3214fbc6a","Type":"ContainerStarted","Data":"a2f1bf7106bc2a1ab8e6ce828bb489028695fe9d5188972ceadb8210881d4d90"} Aug 01 00:59:46.423099 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:46.422936 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 00:59:46.445191 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:46.445133 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" podStartSLOduration=1.445114387 podStartE2EDuration="1.445114387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:59:46.442171762 +0000 UTC m=+1564.651800220" watchObservedRunningTime="2026-08-01 00:59:46.445114387 +0000 UTC m=+1564.654742846" Aug 01 00:59:57.430710 ip-10-0-137-42 kubenswrapper[2568]: I0801 00:59:57.430676 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 01:00:04.905897 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:00:04.905853 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:00:09.092516 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:00:09.092481 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:00:35.893565 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:00:35.893482 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:00:44.999006 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:00:44.998964 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:00:55.899256 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:00:55.899203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:01:00.693815 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:01:00.693773 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:01:09.596555 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:01:09.596523 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:01:16.095174 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:01:16.095132 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:01:21.000389 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:01:21.000357 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:01:30.991194 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:01:30.991158 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:01:36.496570 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:01:36.496538 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:01:46.295408 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:01:46.295373 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:01:53.092955 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:01:53.092915 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:02:02.098719 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:02:02.098684 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:02:26.692709 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:02:26.692670 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:02:31.796600 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:02:31.796560 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:02:39.798353 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:02:39.798322 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:02:54.306923 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:02:54.306887 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:03:29.796178 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:03:29.796134 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:03:38.794969 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:03:38.794885 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:03:45.101579 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:03:45.101541 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:03:55.095547 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:03:55.095506 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:04:02.091494 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:04:02.091458 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:04:09.222013 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:04:09.221973 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:04:19.292888 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:04:19.292850 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:04:26.301883 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:04:26.301834 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:04:38.997876 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:04:38.997843 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:04:44.100871 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:04:44.100835 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:05:01.398854 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:05:01.398822 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:05:06.691499 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:05:06.691408 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:05:20.197641 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:05:20.197604 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:05:25.294984 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:05:25.294943 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:05:31.896523 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:05:31.896483 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:05:36.793248 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:05:36.793191 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:05:42.998962 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:05:42.998922 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:05:52.699596 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:05:52.699563 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:06:02.295916 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:06:02.295880 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:06:20.497535 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:06:20.497497 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:06:27.796886 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:06:27.796849 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:06:37.706249 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:06:37.706121 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:06:53.102517 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:06:53.102477 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:07:00.095282 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:07:00.095242 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:07:11.292596 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:07:11.292561 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:07:16.594356 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:07:16.594312 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:07:26.003465 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:07:26.003428 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:07:33.994401 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:07:33.994366 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:07:48.903903 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:07:48.903865 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:08:10.853112 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:10.853035 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd"] Aug 01 01:08:10.856585 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:10.856570 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" Aug 01 01:08:10.858635 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:10.858613 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:08:10.858751 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:10.858686 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:08:10.863398 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:10.863374 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd"] Aug 01 01:08:10.960924 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:10.960886 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a87ef1-9834-4f3a-bd74-46a302b3b1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd\" (UID: \"62a87ef1-9834-4f3a-bd74-46a302b3b1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" Aug 01 01:08:11.062188 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:11.062152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a87ef1-9834-4f3a-bd74-46a302b3b1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd\" (UID: \"62a87ef1-9834-4f3a-bd74-46a302b3b1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" Aug 01 01:08:11.062862 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:11.062831 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a87ef1-9834-4f3a-bd74-46a302b3b1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd\" (UID: \"62a87ef1-9834-4f3a-bd74-46a302b3b1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" Aug 01 01:08:11.170744 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:11.170658 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" Aug 01 01:08:11.517457 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:11.517424 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd"] Aug 01 01:08:11.519089 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:08:11.519056 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a87ef1_9834_4f3a_bd74_46a302b3b1f5.slice/crio-ba516871da1910610a4e696ff34acb36f2607f10a7e3c442bb0b43b0ca3a510b WatchSource:0}: Error finding container ba516871da1910610a4e696ff34acb36f2607f10a7e3c442bb0b43b0ca3a510b: Status 404 returned error can't find the container with id ba516871da1910610a4e696ff34acb36f2607f10a7e3c442bb0b43b0ca3a510b Aug 01 01:08:11.521285 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:11.521263 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 01:08:12.437929 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:12.437884 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" event={"ID":"62a87ef1-9834-4f3a-bd74-46a302b3b1f5","Type":"ContainerStarted","Data":"ba516871da1910610a4e696ff34acb36f2607f10a7e3c442bb0b43b0ca3a510b"} Aug 01 01:08:14.455884 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:14.455850 2568 generic.go:356] "Generic (PLEG): container finished" podID="62a87ef1-9834-4f3a-bd74-46a302b3b1f5" containerID="ae2de763bea6d1effc2119373da84ff00e6afda46be911251819b662c559421a" exitCode=0 Aug 01 01:08:14.456284 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:14.455937 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" event={"ID":"62a87ef1-9834-4f3a-bd74-46a302b3b1f5","Type":"ContainerDied","Data":"ae2de763bea6d1effc2119373da84ff00e6afda46be911251819b662c559421a"} Aug 01 01:08:15.599115 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:15.599092 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" Aug 01 01:08:15.710850 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:15.710819 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a87ef1-9834-4f3a-bd74-46a302b3b1f5-maas-api-service-ca\") pod \"62a87ef1-9834-4f3a-bd74-46a302b3b1f5\" (UID: \"62a87ef1-9834-4f3a-bd74-46a302b3b1f5\") " Aug 01 01:08:15.711205 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:15.711183 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a87ef1-9834-4f3a-bd74-46a302b3b1f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62a87ef1-9834-4f3a-bd74-46a302b3b1f5" (UID: "62a87ef1-9834-4f3a-bd74-46a302b3b1f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:08:15.811634 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:15.811528 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a87ef1-9834-4f3a-bd74-46a302b3b1f5-maas-api-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:08:16.468433 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:16.468405 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" Aug 01 01:08:16.468601 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:16.468446 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd" event={"ID":"62a87ef1-9834-4f3a-bd74-46a302b3b1f5","Type":"ContainerDied","Data":"ba516871da1910610a4e696ff34acb36f2607f10a7e3c442bb0b43b0ca3a510b"} Aug 01 01:08:16.468601 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:16.468493 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba516871da1910610a4e696ff34acb36f2607f10a7e3c442bb0b43b0ca3a510b" Aug 01 01:08:26.996490 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:26.996455 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd"] Aug 01 01:08:26.998884 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:26.998862 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7ed58d7f-wx5zd"] Aug 01 01:08:28.233999 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:28.233967 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a87ef1-9834-4f3a-bd74-46a302b3b1f5" path="/var/lib/kubelet/pods/62a87ef1-9834-4f3a-bd74-46a302b3b1f5/volumes" Aug 01 01:08:29.899754 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:29.899719 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:08:31.345743 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.345710 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg"] Aug 01 01:08:31.346157 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.346142 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a87ef1-9834-4f3a-bd74-46a302b3b1f5" containerName="revoke-keys" Aug 01 01:08:31.346205 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.346160 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a87ef1-9834-4f3a-bd74-46a302b3b1f5" containerName="revoke-keys" Aug 01 01:08:31.346307 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.346294 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a87ef1-9834-4f3a-bd74-46a302b3b1f5" containerName="revoke-keys" Aug 01 01:08:31.349771 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.349745 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:31.352027 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.352008 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3fd90f77-dockercfg-4k8hn\"" Aug 01 01:08:31.352027 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.352021 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3fd90f77\"" Aug 01 01:08:31.361520 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.361499 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg"] Aug 01 01:08:31.473228 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.473185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszdg\" (UniqueName: \"kubernetes.io/projected/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-kube-api-access-sszdg\") pod \"payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg\" (UID: \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\") " pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:31.473391 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.473353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg\" (UID: \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\") " pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:31.574152 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.574109 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg\" (UID: \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\") " pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:31.574359 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.574182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszdg\" (UniqueName: \"kubernetes.io/projected/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-kube-api-access-sszdg\") pod \"payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg\" (UID: \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\") " pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:31.575096 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.575070 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg\" (UID: \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\") " pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:31.582253 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.582201 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszdg\" (UniqueName: \"kubernetes.io/projected/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-kube-api-access-sszdg\") pod \"payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg\" (UID: \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\") " pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:31.661701 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.661620 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:31.806620 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:31.806594 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg"] Aug 01 01:08:31.809017 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:08:31.808992 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd5163a_b11d_465f_a89a_01f0c4ae3a83.slice/crio-bd32c77b7d66fd11fbfc2423966cf7574abf464b34aff998957d2a899602163b WatchSource:0}: Error finding container bd32c77b7d66fd11fbfc2423966cf7574abf464b34aff998957d2a899602163b: Status 404 returned error can't find the container with id bd32c77b7d66fd11fbfc2423966cf7574abf464b34aff998957d2a899602163b Aug 01 01:08:32.562457 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:32.562404 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" event={"ID":"bfd5163a-b11d-465f-a89a-01f0c4ae3a83","Type":"ContainerStarted","Data":"a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72"} Aug 01 01:08:32.562852 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:32.562481 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" event={"ID":"bfd5163a-b11d-465f-a89a-01f0c4ae3a83","Type":"ContainerStarted","Data":"bd32c77b7d66fd11fbfc2423966cf7574abf464b34aff998957d2a899602163b"} Aug 01 01:08:32.562852 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:32.562515 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:08:32.577589 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:32.577541 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" podStartSLOduration=1.577526725 podStartE2EDuration="1.577526725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:08:32.576458531 +0000 UTC m=+2090.786086991" watchObservedRunningTime="2026-08-01 01:08:32.577526725 +0000 UTC m=+2090.787155182" Aug 01 01:08:37.495833 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:37.495789 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:08:43.571699 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:08:43.571667 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:09:02.048633 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.048601 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg"] Aug 01 01:09:02.049105 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.048873 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" podUID="bfd5163a-b11d-465f-a89a-01f0c4ae3a83" containerName="payload-processing" containerID="cri-o://a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72" gracePeriod=30 Aug 01 01:09:02.321131 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.321069 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:09:02.379515 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.379486 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-plugins-config-volume\") pod \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\" (UID: \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\") " Aug 01 01:09:02.379694 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.379615 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sszdg\" (UniqueName: \"kubernetes.io/projected/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-kube-api-access-sszdg\") pod \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\" (UID: \"bfd5163a-b11d-465f-a89a-01f0c4ae3a83\") " Aug 01 01:09:02.379962 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.379931 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfd5163a-b11d-465f-a89a-01f0c4ae3a83" (UID: "bfd5163a-b11d-465f-a89a-01f0c4ae3a83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:09:02.381823 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.381796 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-kube-api-access-sszdg" (OuterVolumeSpecName: "kube-api-access-sszdg") pod "bfd5163a-b11d-465f-a89a-01f0c4ae3a83" (UID: "bfd5163a-b11d-465f-a89a-01f0c4ae3a83"). InnerVolumeSpecName "kube-api-access-sszdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:09:02.480650 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.480615 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:09:02.480650 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.480646 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sszdg\" (UniqueName: \"kubernetes.io/projected/bfd5163a-b11d-465f-a89a-01f0c4ae3a83-kube-api-access-sszdg\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:09:02.745500 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.745466 2568 generic.go:356] "Generic (PLEG): container finished" podID="bfd5163a-b11d-465f-a89a-01f0c4ae3a83" containerID="a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72" exitCode=0 Aug 01 01:09:02.745700 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.745509 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" Aug 01 01:09:02.745700 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.745522 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" event={"ID":"bfd5163a-b11d-465f-a89a-01f0c4ae3a83","Type":"ContainerDied","Data":"a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72"} Aug 01 01:09:02.745700 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.745577 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg" event={"ID":"bfd5163a-b11d-465f-a89a-01f0c4ae3a83","Type":"ContainerDied","Data":"bd32c77b7d66fd11fbfc2423966cf7574abf464b34aff998957d2a899602163b"} Aug 01 01:09:02.745700 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.745607 2568 scope.go:117] "RemoveContainer" containerID="a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72" Aug 01 01:09:02.755839 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.755816 2568 scope.go:117] "RemoveContainer" containerID="a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72" Aug 01 01:09:02.756086 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:09:02.756067 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72\": container with ID starting with a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72 not found: ID does not exist" containerID="a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72" Aug 01 01:09:02.756138 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.756095 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72"} err="failed to get container status \"a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72\": rpc error: code = NotFound desc = could not find container \"a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72\": container with ID starting with a902e4ac83df6efbba89a800a972d82976719627cb2f7db7b3d535dd52d69e72 not found: ID does not exist" Aug 01 01:09:02.774339 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.774299 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg"] Aug 01 01:09:02.777041 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:02.777015 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3fd90f77-5dfcfd69b5-mbvwg"] Aug 01 01:09:04.231675 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:04.231642 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd5163a-b11d-465f-a89a-01f0c4ae3a83" path="/var/lib/kubelet/pods/bfd5163a-b11d-465f-a89a-01f0c4ae3a83/volumes" Aug 01 01:09:05.101545 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:05.101504 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:09:07.436323 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.436283 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r"] Aug 01 01:09:07.436760 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.436732 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd5163a-b11d-465f-a89a-01f0c4ae3a83" containerName="payload-processing" Aug 01 01:09:07.436760 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.436743 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd5163a-b11d-465f-a89a-01f0c4ae3a83" containerName="payload-processing" Aug 01 01:09:07.436849 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.436817 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd5163a-b11d-465f-a89a-01f0c4ae3a83" containerName="payload-processing" Aug 01 01:09:07.441251 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.441207 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:07.443541 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.443520 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8e664d6c\"" Aug 01 01:09:07.448826 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.448799 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r"] Aug 01 01:09:07.530910 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.530866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r\" (UID: \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:07.531087 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.530921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk76n\" (UniqueName: \"kubernetes.io/projected/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-kube-api-access-zk76n\") pod \"payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r\" (UID: \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:07.632146 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.632105 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk76n\" (UniqueName: \"kubernetes.io/projected/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-kube-api-access-zk76n\") pod \"payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r\" (UID: \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:07.632367 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.632272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r\" (UID: \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:07.632955 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.632936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r\" (UID: \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:07.644871 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.644833 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk76n\" (UniqueName: \"kubernetes.io/projected/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-kube-api-access-zk76n\") pod \"payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r\" (UID: \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:07.753471 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.753368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:07.924641 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:07.924610 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r"] Aug 01 01:09:07.927074 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:09:07.927037 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bd8c79_7a83_4705_98c2_4c661aa0a24a.slice/crio-8e02d904c8b23ac4a51e12ea79040b4ab5a1d453a4ee5cd31c981e807404d0a4 WatchSource:0}: Error finding container 8e02d904c8b23ac4a51e12ea79040b4ab5a1d453a4ee5cd31c981e807404d0a4: Status 404 returned error can't find the container with id 8e02d904c8b23ac4a51e12ea79040b4ab5a1d453a4ee5cd31c981e807404d0a4 Aug 01 01:09:08.785396 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:08.785350 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" event={"ID":"d2bd8c79-7a83-4705-98c2-4c661aa0a24a","Type":"ContainerStarted","Data":"9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5"} Aug 01 01:09:08.785846 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:08.785415 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" event={"ID":"d2bd8c79-7a83-4705-98c2-4c661aa0a24a","Type":"ContainerStarted","Data":"8e02d904c8b23ac4a51e12ea79040b4ab5a1d453a4ee5cd31c981e807404d0a4"} Aug 01 01:09:08.785846 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:08.785511 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:08.801447 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:08.801399 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" podStartSLOduration=1.80138247 podStartE2EDuration="1.80138247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:08.800291985 +0000 UTC m=+2127.009920442" watchObservedRunningTime="2026-08-01 01:09:08.80138247 +0000 UTC m=+2127.011010971" Aug 01 01:09:13.200056 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:13.200015 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:09:19.793984 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:19.793950 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:31.498559 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.498528 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt"] Aug 01 01:09:31.502110 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.502094 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" Aug 01 01:09:31.505100 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.505075 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:09:31.505324 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.505301 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:09:31.509339 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.509316 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt"] Aug 01 01:09:31.568410 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.568380 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7b361-a27f-45b0-9d6a-17600f3c6f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt\" (UID: \"cfe7b361-a27f-45b0-9d6a-17600f3c6f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" Aug 01 01:09:31.669547 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.669507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7b361-a27f-45b0-9d6a-17600f3c6f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt\" (UID: \"cfe7b361-a27f-45b0-9d6a-17600f3c6f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" Aug 01 01:09:31.670114 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.670093 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7b361-a27f-45b0-9d6a-17600f3c6f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt\" (UID: \"cfe7b361-a27f-45b0-9d6a-17600f3c6f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" Aug 01 01:09:31.815258 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.815133 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" Aug 01 01:09:31.954651 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:31.954624 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt"] Aug 01 01:09:31.956311 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:09:31.956280 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe7b361_a27f_45b0_9d6a_17600f3c6f7a.slice/crio-2af6e0e780d12bb707e3907f4c134f4fa81f5c19c04415d0f59a49a89af323c3 WatchSource:0}: Error finding container 2af6e0e780d12bb707e3907f4c134f4fa81f5c19c04415d0f59a49a89af323c3: Status 404 returned error can't find the container with id 2af6e0e780d12bb707e3907f4c134f4fa81f5c19c04415d0f59a49a89af323c3 Aug 01 01:09:32.935069 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:32.935035 2568 generic.go:356] "Generic (PLEG): container finished" podID="cfe7b361-a27f-45b0-9d6a-17600f3c6f7a" containerID="495f4fba0d35ec5c226e1a54460c1ab15f27154a60a7dc42defb38473ee144e7" exitCode=0 Aug 01 01:09:32.935476 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:32.935125 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" event={"ID":"cfe7b361-a27f-45b0-9d6a-17600f3c6f7a","Type":"ContainerDied","Data":"495f4fba0d35ec5c226e1a54460c1ab15f27154a60a7dc42defb38473ee144e7"} Aug 01 01:09:32.935476 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:32.935242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" event={"ID":"cfe7b361-a27f-45b0-9d6a-17600f3c6f7a","Type":"ContainerStarted","Data":"2af6e0e780d12bb707e3907f4c134f4fa81f5c19c04415d0f59a49a89af323c3"} Aug 01 01:09:34.084225 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:34.084184 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" Aug 01 01:09:34.196335 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:34.196204 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7b361-a27f-45b0-9d6a-17600f3c6f7a-maas-api-service-ca\") pod \"cfe7b361-a27f-45b0-9d6a-17600f3c6f7a\" (UID: \"cfe7b361-a27f-45b0-9d6a-17600f3c6f7a\") " Aug 01 01:09:34.196556 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:34.196535 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe7b361-a27f-45b0-9d6a-17600f3c6f7a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfe7b361-a27f-45b0-9d6a-17600f3c6f7a" (UID: "cfe7b361-a27f-45b0-9d6a-17600f3c6f7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:09:34.297565 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:34.297529 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe7b361-a27f-45b0-9d6a-17600f3c6f7a-maas-api-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:09:34.948763 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:34.948733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" event={"ID":"cfe7b361-a27f-45b0-9d6a-17600f3c6f7a","Type":"ContainerDied","Data":"2af6e0e780d12bb707e3907f4c134f4fa81f5c19c04415d0f59a49a89af323c3"} Aug 01 01:09:34.948763 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:34.948762 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt" Aug 01 01:09:34.948982 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:34.948784 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af6e0e780d12bb707e3907f4c134f4fa81f5c19c04415d0f59a49a89af323c3" Aug 01 01:09:37.454912 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.454879 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r"] Aug 01 01:09:37.455514 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.455120 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" podUID="d2bd8c79-7a83-4705-98c2-4c661aa0a24a" containerName="payload-pre-processing" containerID="cri-o://9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5" gracePeriod=30 Aug 01 01:09:37.604652 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.604622 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt"] Aug 01 01:09:37.607423 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.607400 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e664d6c-65tmt"] Aug 01 01:09:37.724261 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.724237 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:37.831691 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.831657 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-plugins-config-volume\") pod \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\" (UID: \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\") " Aug 01 01:09:37.831855 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.831711 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk76n\" (UniqueName: \"kubernetes.io/projected/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-kube-api-access-zk76n\") pod \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\" (UID: \"d2bd8c79-7a83-4705-98c2-4c661aa0a24a\") " Aug 01 01:09:37.832030 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.832007 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2bd8c79-7a83-4705-98c2-4c661aa0a24a" (UID: "d2bd8c79-7a83-4705-98c2-4c661aa0a24a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:09:37.833931 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.833911 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-kube-api-access-zk76n" (OuterVolumeSpecName: "kube-api-access-zk76n") pod "d2bd8c79-7a83-4705-98c2-4c661aa0a24a" (UID: "d2bd8c79-7a83-4705-98c2-4c661aa0a24a"). InnerVolumeSpecName "kube-api-access-zk76n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:09:37.933154 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.933117 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:09:37.933154 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.933147 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk76n\" (UniqueName: \"kubernetes.io/projected/d2bd8c79-7a83-4705-98c2-4c661aa0a24a-kube-api-access-zk76n\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:09:37.967565 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.967483 2568 generic.go:356] "Generic (PLEG): container finished" podID="d2bd8c79-7a83-4705-98c2-4c661aa0a24a" containerID="9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5" exitCode=0 Aug 01 01:09:37.967565 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.967540 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" Aug 01 01:09:37.967732 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.967542 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" event={"ID":"d2bd8c79-7a83-4705-98c2-4c661aa0a24a","Type":"ContainerDied","Data":"9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5"} Aug 01 01:09:37.967732 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.967678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r" event={"ID":"d2bd8c79-7a83-4705-98c2-4c661aa0a24a","Type":"ContainerDied","Data":"8e02d904c8b23ac4a51e12ea79040b4ab5a1d453a4ee5cd31c981e807404d0a4"} Aug 01 01:09:37.967732 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.967702 2568 scope.go:117] "RemoveContainer" containerID="9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5" Aug 01 01:09:37.978092 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.978076 2568 scope.go:117] "RemoveContainer" containerID="9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5" Aug 01 01:09:37.978386 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:09:37.978367 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5\": container with ID starting with 9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5 not found: ID does not exist" containerID="9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5" Aug 01 01:09:37.978454 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.978397 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5"} err="failed to get container status \"9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5\": rpc error: code = NotFound desc = could not find container \"9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5\": container with ID starting with 9f02137dff55704c58500e49d727c76bd1d706d78c79061ecd977a2014f38bc5 not found: ID does not exist" Aug 01 01:09:37.990324 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.990299 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r"] Aug 01 01:09:37.996176 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:37.996155 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8e664d6c-6478df7f8d-ds47r"] Aug 01 01:09:38.235346 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:38.235264 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe7b361-a27f-45b0-9d6a-17600f3c6f7a" path="/var/lib/kubelet/pods/cfe7b361-a27f-45b0-9d6a-17600f3c6f7a/volumes" Aug 01 01:09:38.235638 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:38.235624 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2bd8c79-7a83-4705-98c2-4c661aa0a24a" path="/var/lib/kubelet/pods/d2bd8c79-7a83-4705-98c2-4c661aa0a24a/volumes" Aug 01 01:09:40.755787 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:40.755750 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:09:43.583047 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.583010 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6"] Aug 01 01:09:43.583502 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.583487 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe7b361-a27f-45b0-9d6a-17600f3c6f7a" containerName="revoke-keys" Aug 01 01:09:43.583548 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.583504 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe7b361-a27f-45b0-9d6a-17600f3c6f7a" containerName="revoke-keys" Aug 01 01:09:43.583548 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.583515 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2bd8c79-7a83-4705-98c2-4c661aa0a24a" containerName="payload-pre-processing" Aug 01 01:09:43.583548 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.583521 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2bd8c79-7a83-4705-98c2-4c661aa0a24a" containerName="payload-pre-processing" Aug 01 01:09:43.583683 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.583595 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2bd8c79-7a83-4705-98c2-4c661aa0a24a" containerName="payload-pre-processing" Aug 01 01:09:43.583683 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.583603 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe7b361-a27f-45b0-9d6a-17600f3c6f7a" containerName="revoke-keys" Aug 01 01:09:43.588165 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.588142 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:43.590444 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.590411 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ef000e4f\"" Aug 01 01:09:43.590546 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.590417 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ef000e4f-dockercfg-g9g2l\"" Aug 01 01:09:43.595572 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.595548 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6"] Aug 01 01:09:43.624239 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.622164 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt"] Aug 01 01:09:43.627853 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.626756 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:43.634576 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.634548 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt"] Aug 01 01:09:43.690136 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.690088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ddf585-62d5-4550-bdce-029b159d769c-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt\" (UID: \"49ddf585-62d5-4550-bdce-029b159d769c\") " pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:43.690408 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.690148 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpd2p\" (UniqueName: \"kubernetes.io/projected/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-kube-api-access-hpd2p\") pod \"payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6\" (UID: \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:43.690408 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.690333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6\" (UID: \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:43.690408 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.690379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkd2\" (UniqueName: \"kubernetes.io/projected/49ddf585-62d5-4550-bdce-029b159d769c-kube-api-access-6nkd2\") pod \"payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt\" (UID: \"49ddf585-62d5-4550-bdce-029b159d769c\") " pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:43.790912 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.790868 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ddf585-62d5-4550-bdce-029b159d769c-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt\" (UID: \"49ddf585-62d5-4550-bdce-029b159d769c\") " pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:43.791105 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.790923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpd2p\" (UniqueName: \"kubernetes.io/projected/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-kube-api-access-hpd2p\") pod \"payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6\" (UID: \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:43.791105 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.791022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6\" (UID: \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:43.791105 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.791057 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkd2\" (UniqueName: \"kubernetes.io/projected/49ddf585-62d5-4550-bdce-029b159d769c-kube-api-access-6nkd2\") pod \"payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt\" (UID: \"49ddf585-62d5-4550-bdce-029b159d769c\") " pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:43.791692 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.791671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ddf585-62d5-4550-bdce-029b159d769c-plugins-config-volume\") pod \"payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt\" (UID: \"49ddf585-62d5-4550-bdce-029b159d769c\") " pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:43.791760 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.791741 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6\" (UID: \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:43.798407 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.798381 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpd2p\" (UniqueName: \"kubernetes.io/projected/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-kube-api-access-hpd2p\") pod \"payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6\" (UID: \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:43.798538 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.798492 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkd2\" (UniqueName: \"kubernetes.io/projected/49ddf585-62d5-4550-bdce-029b159d769c-kube-api-access-6nkd2\") pod \"payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt\" (UID: \"49ddf585-62d5-4550-bdce-029b159d769c\") " pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:43.900650 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.900547 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:43.942352 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:43.942326 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:44.058126 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:44.058095 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6"] Aug 01 01:09:44.061353 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:09:44.061317 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8602e4fc_54ba_4fa1_9ea1_e1aa3ffe8e6b.slice/crio-334a245c37c7a024ae75b5d2fad748d805b4d85081be013d40c975c4d069fa7a WatchSource:0}: Error finding container 334a245c37c7a024ae75b5d2fad748d805b4d85081be013d40c975c4d069fa7a: Status 404 returned error can't find the container with id 334a245c37c7a024ae75b5d2fad748d805b4d85081be013d40c975c4d069fa7a Aug 01 01:09:44.109107 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:44.109085 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt"] Aug 01 01:09:44.111179 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:09:44.111148 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ddf585_62d5_4550_bdce_029b159d769c.slice/crio-045da2a264ae69333f4c307a5980c7b1ac34df4587a9dea2efa3ed507ccef192 WatchSource:0}: Error finding container 045da2a264ae69333f4c307a5980c7b1ac34df4587a9dea2efa3ed507ccef192: Status 404 returned error can't find the container with id 045da2a264ae69333f4c307a5980c7b1ac34df4587a9dea2efa3ed507ccef192 Aug 01 01:09:45.016236 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:45.016182 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" event={"ID":"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b","Type":"ContainerStarted","Data":"1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505"} Aug 01 01:09:45.016734 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:45.016264 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:09:45.016734 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:45.016286 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" event={"ID":"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b","Type":"ContainerStarted","Data":"334a245c37c7a024ae75b5d2fad748d805b4d85081be013d40c975c4d069fa7a"} Aug 01 01:09:45.017721 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:45.017697 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" event={"ID":"49ddf585-62d5-4550-bdce-029b159d769c","Type":"ContainerStarted","Data":"c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de"} Aug 01 01:09:45.017803 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:45.017745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" event={"ID":"49ddf585-62d5-4550-bdce-029b159d769c","Type":"ContainerStarted","Data":"045da2a264ae69333f4c307a5980c7b1ac34df4587a9dea2efa3ed507ccef192"} Aug 01 01:09:45.017803 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:45.017769 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:45.032354 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:45.032304 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" podStartSLOduration=2.032286197 podStartE2EDuration="2.032286197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:45.029246849 +0000 UTC m=+2163.238875302" watchObservedRunningTime="2026-08-01 01:09:45.032286197 +0000 UTC m=+2163.241914656" Aug 01 01:09:45.045649 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:45.045605 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" podStartSLOduration=2.045591958 podStartE2EDuration="2.045591958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:09:45.042980817 +0000 UTC m=+2163.252609285" watchObservedRunningTime="2026-08-01 01:09:45.045591958 +0000 UTC m=+2163.255220415" Aug 01 01:09:49.087129 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:49.087039 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:09:56.025608 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:56.025572 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:09:56.026120 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:09:56.025664 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:10:13.550416 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.550383 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt"] Aug 01 01:10:13.551336 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.550674 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" podUID="49ddf585-62d5-4550-bdce-029b159d769c" containerName="payload-processing" containerID="cri-o://c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de" gracePeriod=30 Aug 01 01:10:13.557574 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.557540 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6"] Aug 01 01:10:13.557996 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.557968 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" podUID="8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b" containerName="payload-pre-processing" containerID="cri-o://1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505" gracePeriod=30 Aug 01 01:10:13.840630 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.840589 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:10:13.875090 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.875059 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ddf585-62d5-4550-bdce-029b159d769c-plugins-config-volume\") pod \"49ddf585-62d5-4550-bdce-029b159d769c\" (UID: \"49ddf585-62d5-4550-bdce-029b159d769c\") " Aug 01 01:10:13.875243 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.875185 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nkd2\" (UniqueName: \"kubernetes.io/projected/49ddf585-62d5-4550-bdce-029b159d769c-kube-api-access-6nkd2\") pod \"49ddf585-62d5-4550-bdce-029b159d769c\" (UID: \"49ddf585-62d5-4550-bdce-029b159d769c\") " Aug 01 01:10:13.875479 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.875453 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ddf585-62d5-4550-bdce-029b159d769c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49ddf585-62d5-4550-bdce-029b159d769c" (UID: "49ddf585-62d5-4550-bdce-029b159d769c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:10:13.875570 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.875545 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ddf585-62d5-4550-bdce-029b159d769c-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:10:13.877553 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.877530 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ddf585-62d5-4550-bdce-029b159d769c-kube-api-access-6nkd2" (OuterVolumeSpecName: "kube-api-access-6nkd2") pod "49ddf585-62d5-4550-bdce-029b159d769c" (UID: "49ddf585-62d5-4550-bdce-029b159d769c"). InnerVolumeSpecName "kube-api-access-6nkd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:10:13.878966 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.878946 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:10:13.976722 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.976691 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpd2p\" (UniqueName: \"kubernetes.io/projected/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-kube-api-access-hpd2p\") pod \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\" (UID: \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\") " Aug 01 01:10:13.976888 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.976841 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-plugins-config-volume\") pod \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\" (UID: \"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b\") " Aug 01 01:10:13.977124 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.977106 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkd2\" (UniqueName: \"kubernetes.io/projected/49ddf585-62d5-4550-bdce-029b159d769c-kube-api-access-6nkd2\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:10:13.977186 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.977164 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b" (UID: "8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:10:13.979004 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:13.978983 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-kube-api-access-hpd2p" (OuterVolumeSpecName: "kube-api-access-hpd2p") pod "8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b" (UID: "8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b"). InnerVolumeSpecName "kube-api-access-hpd2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:10:14.078563 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.078489 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:10:14.078563 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.078517 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpd2p\" (UniqueName: \"kubernetes.io/projected/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b-kube-api-access-hpd2p\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:10:14.193626 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.193593 2568 generic.go:356] "Generic (PLEG): container finished" podID="8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b" containerID="1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505" exitCode=0 Aug 01 01:10:14.193792 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.193636 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" Aug 01 01:10:14.193792 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.193678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" event={"ID":"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b","Type":"ContainerDied","Data":"1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505"} Aug 01 01:10:14.193792 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.193726 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6" event={"ID":"8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b","Type":"ContainerDied","Data":"334a245c37c7a024ae75b5d2fad748d805b4d85081be013d40c975c4d069fa7a"} Aug 01 01:10:14.193792 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.193752 2568 scope.go:117] "RemoveContainer" containerID="1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505" Aug 01 01:10:14.195006 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.194980 2568 generic.go:356] "Generic (PLEG): container finished" podID="49ddf585-62d5-4550-bdce-029b159d769c" containerID="c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de" exitCode=0 Aug 01 01:10:14.195131 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.195017 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" Aug 01 01:10:14.195131 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.195063 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" event={"ID":"49ddf585-62d5-4550-bdce-029b159d769c","Type":"ContainerDied","Data":"c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de"} Aug 01 01:10:14.195131 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.195113 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt" event={"ID":"49ddf585-62d5-4550-bdce-029b159d769c","Type":"ContainerDied","Data":"045da2a264ae69333f4c307a5980c7b1ac34df4587a9dea2efa3ed507ccef192"} Aug 01 01:10:14.204057 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.204040 2568 scope.go:117] "RemoveContainer" containerID="1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505" Aug 01 01:10:14.204381 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:10:14.204350 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505\": container with ID starting with 1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505 not found: ID does not exist" containerID="1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505" Aug 01 01:10:14.204519 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.204393 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505"} err="failed to get container status \"1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505\": rpc error: code = NotFound desc = could not find container \"1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505\": container with ID starting with 1fcf125479475db25f59ee3bffde015cfeecfe2b8ae354af94e7e3692cf54505 not found: ID does not exist" Aug 01 01:10:14.204519 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.204417 2568 scope.go:117] "RemoveContainer" containerID="c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de" Aug 01 01:10:14.216504 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.216483 2568 scope.go:117] "RemoveContainer" containerID="c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de" Aug 01 01:10:14.216784 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:10:14.216759 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de\": container with ID starting with c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de not found: ID does not exist" containerID="c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de" Aug 01 01:10:14.216888 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.216791 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de"} err="failed to get container status \"c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de\": rpc error: code = NotFound desc = could not find container \"c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de\": container with ID starting with c5b723504cb1615fb28479e7b0d694c9fc8aedc391426764253a66b6560339de not found: ID does not exist" Aug 01 01:10:14.219956 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.219936 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt"] Aug 01 01:10:14.223092 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.223072 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ef000e4f-5c95c9bbf8-9nbnt"] Aug 01 01:10:14.231243 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.231201 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ddf585-62d5-4550-bdce-029b159d769c" path="/var/lib/kubelet/pods/49ddf585-62d5-4550-bdce-029b159d769c/volumes" Aug 01 01:10:14.232877 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.232859 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6"] Aug 01 01:10:14.236259 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:14.236239 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef000e4f-7b699f4c44-rd9m6"] Aug 01 01:10:16.230386 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:16.230354 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b" path="/var/lib/kubelet/pods/8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b/volumes" Aug 01 01:10:16.602272 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:16.602165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:10:29.392839 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:10:29.392801 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:11:57.003616 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:11:57.003579 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:12:05.098232 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:05.098175 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:12:17.290490 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:17.290456 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:12:45.573822 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.573746 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st"] Aug 01 01:12:45.574375 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.574207 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b" containerName="payload-pre-processing" Aug 01 01:12:45.574375 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.574233 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b" containerName="payload-pre-processing" Aug 01 01:12:45.574375 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.574270 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ddf585-62d5-4550-bdce-029b159d769c" containerName="payload-processing" Aug 01 01:12:45.574375 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.574276 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ddf585-62d5-4550-bdce-029b159d769c" containerName="payload-processing" Aug 01 01:12:45.574375 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.574346 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ddf585-62d5-4550-bdce-029b159d769c" containerName="payload-processing" Aug 01 01:12:45.574375 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.574358 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8602e4fc-54ba-4fa1-9ea1-e1aa3ffe8e6b" containerName="payload-pre-processing" Aug 01 01:12:45.577632 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.577617 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" Aug 01 01:12:45.580078 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.580054 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:12:45.580492 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.580462 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:12:45.583724 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.583695 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st"] Aug 01 01:12:45.656310 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.656277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4756e52b-725f-49ca-ab7d-4f636f750202-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86872dcf-7k8st\" (UID: \"4756e52b-725f-49ca-ab7d-4f636f750202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" Aug 01 01:12:45.757528 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.757486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4756e52b-725f-49ca-ab7d-4f636f750202-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86872dcf-7k8st\" (UID: \"4756e52b-725f-49ca-ab7d-4f636f750202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" Aug 01 01:12:45.758099 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.758079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4756e52b-725f-49ca-ab7d-4f636f750202-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86872dcf-7k8st\" (UID: \"4756e52b-725f-49ca-ab7d-4f636f750202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" Aug 01 01:12:45.891920 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:45.891835 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" Aug 01 01:12:46.036858 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:46.036831 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st"] Aug 01 01:12:46.038710 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:12:46.038683 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4756e52b_725f_49ca_ab7d_4f636f750202.slice/crio-bff190db2a8a4f8b76c8b4ed56fb548ee494d4cb07f6023259ab46d6c8bb0415 WatchSource:0}: Error finding container bff190db2a8a4f8b76c8b4ed56fb548ee494d4cb07f6023259ab46d6c8bb0415: Status 404 returned error can't find the container with id bff190db2a8a4f8b76c8b4ed56fb548ee494d4cb07f6023259ab46d6c8bb0415 Aug 01 01:12:46.111567 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:46.111531 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" event={"ID":"4756e52b-725f-49ca-ab7d-4f636f750202","Type":"ContainerStarted","Data":"bff190db2a8a4f8b76c8b4ed56fb548ee494d4cb07f6023259ab46d6c8bb0415"} Aug 01 01:12:47.119126 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:47.119090 2568 generic.go:356] "Generic (PLEG): container finished" podID="4756e52b-725f-49ca-ab7d-4f636f750202" containerID="d442ce7afecad5ed93d290a098919518326749d3fe5e9409d3ef11e0aa207f5e" exitCode=0 Aug 01 01:12:47.119550 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:47.119191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" event={"ID":"4756e52b-725f-49ca-ab7d-4f636f750202","Type":"ContainerDied","Data":"d442ce7afecad5ed93d290a098919518326749d3fe5e9409d3ef11e0aa207f5e"} Aug 01 01:12:48.267243 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:48.267198 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" Aug 01 01:12:48.382854 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:48.382768 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4756e52b-725f-49ca-ab7d-4f636f750202-maas-api-service-ca\") pod \"4756e52b-725f-49ca-ab7d-4f636f750202\" (UID: \"4756e52b-725f-49ca-ab7d-4f636f750202\") " Aug 01 01:12:48.383125 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:48.383103 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4756e52b-725f-49ca-ab7d-4f636f750202-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4756e52b-725f-49ca-ab7d-4f636f750202" (UID: "4756e52b-725f-49ca-ab7d-4f636f750202"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:12:48.483527 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:48.483491 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4756e52b-725f-49ca-ab7d-4f636f750202-maas-api-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:12:49.133108 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:49.133075 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" Aug 01 01:12:49.133108 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:49.133104 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st" event={"ID":"4756e52b-725f-49ca-ab7d-4f636f750202","Type":"ContainerDied","Data":"bff190db2a8a4f8b76c8b4ed56fb548ee494d4cb07f6023259ab46d6c8bb0415"} Aug 01 01:12:49.133350 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:49.133148 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff190db2a8a4f8b76c8b4ed56fb548ee494d4cb07f6023259ab46d6c8bb0415" Aug 01 01:12:56.120497 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:56.120450 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st"] Aug 01 01:12:56.123051 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:56.123028 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86872dcf-7k8st"] Aug 01 01:12:56.231130 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:56.231100 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4756e52b-725f-49ca-ab7d-4f636f750202" path="/var/lib/kubelet/pods/4756e52b-725f-49ca-ab7d-4f636f750202/volumes" Aug 01 01:12:58.000187 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:12:58.000145 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:13:36.601288 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:36.601252 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:13:43.575676 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:43.575641 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm"] Aug 01 01:13:43.576156 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:43.575862 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" podUID="ba879383-aa83-42bd-887f-8ff3214fbc6a" containerName="manager" containerID="cri-o://7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1" gracePeriod=10 Aug 01 01:13:43.837442 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:43.837373 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 01:13:43.921588 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:43.921547 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba879383-aa83-42bd-887f-8ff3214fbc6a-extensions-socket-volume\") pod \"ba879383-aa83-42bd-887f-8ff3214fbc6a\" (UID: \"ba879383-aa83-42bd-887f-8ff3214fbc6a\") " Aug 01 01:13:43.921767 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:43.921638 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l2xd\" (UniqueName: \"kubernetes.io/projected/ba879383-aa83-42bd-887f-8ff3214fbc6a-kube-api-access-2l2xd\") pod \"ba879383-aa83-42bd-887f-8ff3214fbc6a\" (UID: \"ba879383-aa83-42bd-887f-8ff3214fbc6a\") " Aug 01 01:13:43.921923 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:43.921897 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba879383-aa83-42bd-887f-8ff3214fbc6a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ba879383-aa83-42bd-887f-8ff3214fbc6a" (UID: "ba879383-aa83-42bd-887f-8ff3214fbc6a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:13:43.923864 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:43.923829 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba879383-aa83-42bd-887f-8ff3214fbc6a-kube-api-access-2l2xd" (OuterVolumeSpecName: "kube-api-access-2l2xd") pod "ba879383-aa83-42bd-887f-8ff3214fbc6a" (UID: "ba879383-aa83-42bd-887f-8ff3214fbc6a"). InnerVolumeSpecName "kube-api-access-2l2xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:13:44.022670 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.022630 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba879383-aa83-42bd-887f-8ff3214fbc6a-extensions-socket-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:13:44.022670 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.022670 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l2xd\" (UniqueName: \"kubernetes.io/projected/ba879383-aa83-42bd-887f-8ff3214fbc6a-kube-api-access-2l2xd\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:13:44.460346 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.460304 2568 generic.go:356] "Generic (PLEG): container finished" podID="ba879383-aa83-42bd-887f-8ff3214fbc6a" containerID="7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1" exitCode=0 Aug 01 01:13:44.460530 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.460355 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" Aug 01 01:13:44.460530 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.460387 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" event={"ID":"ba879383-aa83-42bd-887f-8ff3214fbc6a","Type":"ContainerDied","Data":"7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1"} Aug 01 01:13:44.460530 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.460438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm" event={"ID":"ba879383-aa83-42bd-887f-8ff3214fbc6a","Type":"ContainerDied","Data":"a2f1bf7106bc2a1ab8e6ce828bb489028695fe9d5188972ceadb8210881d4d90"} Aug 01 01:13:44.460530 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.460459 2568 scope.go:117] "RemoveContainer" containerID="7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1" Aug 01 01:13:44.471810 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.471785 2568 scope.go:117] "RemoveContainer" containerID="7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1" Aug 01 01:13:44.472111 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:13:44.472084 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1\": container with ID starting with 7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1 not found: ID does not exist" containerID="7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1" Aug 01 01:13:44.472163 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.472125 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1"} err="failed to get container status \"7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1\": rpc error: code = NotFound desc = could not find container \"7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1\": container with ID starting with 7b463967a28a66cabfe7e27b3be047987127877c69dbffb054b549b3cba159a1 not found: ID does not exist" Aug 01 01:13:44.477273 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.477245 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm"] Aug 01 01:13:44.479695 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:44.479671 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gw6qm"] Aug 01 01:13:46.231105 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:13:46.231069 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba879383-aa83-42bd-887f-8ff3214fbc6a" path="/var/lib/kubelet/pods/ba879383-aa83-42bd-887f-8ff3214fbc6a/volumes" Aug 01 01:14:42.509162 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:42.509128 2568 scope.go:117] "RemoveContainer" containerID="ae2de763bea6d1effc2119373da84ff00e6afda46be911251819b662c559421a" Aug 01 01:14:59.214662 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.214626 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv"] Aug 01 01:14:59.215159 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.215045 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4756e52b-725f-49ca-ab7d-4f636f750202" containerName="revoke-keys" Aug 01 01:14:59.215159 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.215056 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4756e52b-725f-49ca-ab7d-4f636f750202" containerName="revoke-keys" Aug 01 01:14:59.215159 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.215070 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba879383-aa83-42bd-887f-8ff3214fbc6a" containerName="manager" Aug 01 01:14:59.215159 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.215075 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba879383-aa83-42bd-887f-8ff3214fbc6a" containerName="manager" Aug 01 01:14:59.215159 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.215151 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba879383-aa83-42bd-887f-8ff3214fbc6a" containerName="manager" Aug 01 01:14:59.215159 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.215158 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4756e52b-725f-49ca-ab7d-4f636f750202" containerName="revoke-keys" Aug 01 01:14:59.218311 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.218294 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.220986 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.220965 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8zn4g\"" Aug 01 01:14:59.228808 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.228784 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv"] Aug 01 01:14:59.342895 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.342848 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a269c374-4cdd-445f-8459-2c71eb088c98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-th6rv\" (UID: \"a269c374-4cdd-445f-8459-2c71eb088c98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.342895 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.342900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdc4r\" (UniqueName: \"kubernetes.io/projected/a269c374-4cdd-445f-8459-2c71eb088c98-kube-api-access-tdc4r\") pod \"kuadrant-operator-controller-manager-68cf589589-th6rv\" (UID: \"a269c374-4cdd-445f-8459-2c71eb088c98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.444045 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.443991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a269c374-4cdd-445f-8459-2c71eb088c98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-th6rv\" (UID: \"a269c374-4cdd-445f-8459-2c71eb088c98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.444045 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.444053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdc4r\" (UniqueName: \"kubernetes.io/projected/a269c374-4cdd-445f-8459-2c71eb088c98-kube-api-access-tdc4r\") pod \"kuadrant-operator-controller-manager-68cf589589-th6rv\" (UID: \"a269c374-4cdd-445f-8459-2c71eb088c98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.444498 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.444475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a269c374-4cdd-445f-8459-2c71eb088c98-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-th6rv\" (UID: \"a269c374-4cdd-445f-8459-2c71eb088c98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.452411 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.452386 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdc4r\" (UniqueName: \"kubernetes.io/projected/a269c374-4cdd-445f-8459-2c71eb088c98-kube-api-access-tdc4r\") pod \"kuadrant-operator-controller-manager-68cf589589-th6rv\" (UID: \"a269c374-4cdd-445f-8459-2c71eb088c98\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.530076 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.529985 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.679196 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.678999 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv"] Aug 01 01:14:59.682623 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:14:59.682594 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda269c374_4cdd_445f_8459_2c71eb088c98.slice/crio-acc565a2fcf81c84da0821d4865e75145713891eed857802986d38b8faa7a17f WatchSource:0}: Error finding container acc565a2fcf81c84da0821d4865e75145713891eed857802986d38b8faa7a17f: Status 404 returned error can't find the container with id acc565a2fcf81c84da0821d4865e75145713891eed857802986d38b8faa7a17f Aug 01 01:14:59.905776 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.905681 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" event={"ID":"a269c374-4cdd-445f-8459-2c71eb088c98","Type":"ContainerStarted","Data":"bef1bd432a497f86c56e8d74879b350efa7bc538e62d5bf99c0b4fa75a7992c6"} Aug 01 01:14:59.905776 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.905746 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:14:59.905776 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.905766 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" event={"ID":"a269c374-4cdd-445f-8459-2c71eb088c98","Type":"ContainerStarted","Data":"acc565a2fcf81c84da0821d4865e75145713891eed857802986d38b8faa7a17f"} Aug 01 01:14:59.924705 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:14:59.924653 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" podStartSLOduration=0.924638038 podStartE2EDuration="924.638038ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:14:59.921409484 +0000 UTC m=+2478.131037966" watchObservedRunningTime="2026-08-01 01:14:59.924638038 +0000 UTC m=+2478.134266495" Aug 01 01:15:10.917966 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:10.917931 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-th6rv" Aug 01 01:15:19.298459 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:19.298406 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:15:22.805472 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:22.805437 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn"] Aug 01 01:15:22.809430 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:22.809412 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:22.811630 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:22.811603 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ff21edf2\"" Aug 01 01:15:22.811746 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:22.811685 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ff21edf2-dockercfg-25m55\"" Aug 01 01:15:22.818404 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:22.818378 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn"] Aug 01 01:15:22.983199 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:22.983166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387706a-2efd-42e2-90ea-2eb92bc4e07b-plugins-config-volume\") pod \"payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn\" (UID: \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\") " pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:22.983387 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:22.983204 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfj2c\" (UniqueName: \"kubernetes.io/projected/2387706a-2efd-42e2-90ea-2eb92bc4e07b-kube-api-access-zfj2c\") pod \"payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn\" (UID: \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\") " pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:23.084348 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:23.084259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387706a-2efd-42e2-90ea-2eb92bc4e07b-plugins-config-volume\") pod \"payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn\" (UID: \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\") " pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:23.084348 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:23.084297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfj2c\" (UniqueName: \"kubernetes.io/projected/2387706a-2efd-42e2-90ea-2eb92bc4e07b-kube-api-access-zfj2c\") pod \"payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn\" (UID: \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\") " pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:23.084956 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:23.084937 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387706a-2efd-42e2-90ea-2eb92bc4e07b-plugins-config-volume\") pod \"payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn\" (UID: \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\") " pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:23.095120 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:23.095094 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfj2c\" (UniqueName: \"kubernetes.io/projected/2387706a-2efd-42e2-90ea-2eb92bc4e07b-kube-api-access-zfj2c\") pod \"payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn\" (UID: \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\") " pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:23.120808 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:23.120770 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:23.271757 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:23.271731 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn"] Aug 01 01:15:23.273428 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:15:23.273403 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2387706a_2efd_42e2_90ea_2eb92bc4e07b.slice/crio-4985ccebcff160b82aede38f2517bb23429ab86d7b456d76995ea12686f1d5eb WatchSource:0}: Error finding container 4985ccebcff160b82aede38f2517bb23429ab86d7b456d76995ea12686f1d5eb: Status 404 returned error can't find the container with id 4985ccebcff160b82aede38f2517bb23429ab86d7b456d76995ea12686f1d5eb Aug 01 01:15:24.057669 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:24.057631 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" event={"ID":"2387706a-2efd-42e2-90ea-2eb92bc4e07b","Type":"ContainerStarted","Data":"8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c"} Aug 01 01:15:24.058090 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:24.057705 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" event={"ID":"2387706a-2efd-42e2-90ea-2eb92bc4e07b","Type":"ContainerStarted","Data":"4985ccebcff160b82aede38f2517bb23429ab86d7b456d76995ea12686f1d5eb"} Aug 01 01:15:24.058090 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:24.057836 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:24.072168 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:24.072113 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" podStartSLOduration=2.072098041 podStartE2EDuration="2.072098041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:15:24.070767729 +0000 UTC m=+2502.280396190" watchObservedRunningTime="2026-08-01 01:15:24.072098041 +0000 UTC m=+2502.281726497" Aug 01 01:15:27.694579 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:27.694544 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:15:35.067475 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:35.067374 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:15:42.540461 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:15:42.540424 2568 scope.go:117] "RemoveContainer" containerID="495f4fba0d35ec5c226e1a54460c1ab15f27154a60a7dc42defb38473ee144e7" Aug 01 01:16:03.199166 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:03.199126 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:16:16.794341 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:16.794302 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:16:25.833523 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:25.833482 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j"] Aug 01 01:16:25.837230 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:25.837196 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" Aug 01 01:16:25.839281 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:25.839256 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:16:25.839421 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:25.839296 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:16:25.844386 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:25.844339 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j"] Aug 01 01:16:25.902305 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:25.902261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/363fcebf-c7da-4d18-8541-538a076f0d94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j\" (UID: \"363fcebf-c7da-4d18-8541-538a076f0d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" Aug 01 01:16:26.003804 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:26.003767 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/363fcebf-c7da-4d18-8541-538a076f0d94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j\" (UID: \"363fcebf-c7da-4d18-8541-538a076f0d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" Aug 01 01:16:26.004458 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:26.004438 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/363fcebf-c7da-4d18-8541-538a076f0d94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j\" (UID: \"363fcebf-c7da-4d18-8541-538a076f0d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" Aug 01 01:16:26.149783 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:26.149692 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" Aug 01 01:16:26.496002 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:26.495972 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j"] Aug 01 01:16:26.497755 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:16:26.497731 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363fcebf_c7da_4d18_8541_538a076f0d94.slice/crio-75757dd71ecfcde2ec15ba62c9676cc89f137ef18a7184ddb4c6b9801e0ca5e8 WatchSource:0}: Error finding container 75757dd71ecfcde2ec15ba62c9676cc89f137ef18a7184ddb4c6b9801e0ca5e8: Status 404 returned error can't find the container with id 75757dd71ecfcde2ec15ba62c9676cc89f137ef18a7184ddb4c6b9801e0ca5e8 Aug 01 01:16:27.444764 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:27.444727 2568 generic.go:356] "Generic (PLEG): container finished" podID="363fcebf-c7da-4d18-8541-538a076f0d94" containerID="3a329aef8fb0adc0d78a62d554c1065a89e2072e99c9dc26a6c87725a9e6f33e" exitCode=0 Aug 01 01:16:27.445139 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:27.444811 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" event={"ID":"363fcebf-c7da-4d18-8541-538a076f0d94","Type":"ContainerDied","Data":"3a329aef8fb0adc0d78a62d554c1065a89e2072e99c9dc26a6c87725a9e6f33e"} Aug 01 01:16:27.445139 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:27.444861 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" event={"ID":"363fcebf-c7da-4d18-8541-538a076f0d94","Type":"ContainerStarted","Data":"75757dd71ecfcde2ec15ba62c9676cc89f137ef18a7184ddb4c6b9801e0ca5e8"} Aug 01 01:16:28.594468 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:28.594434 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" Aug 01 01:16:28.730516 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:28.730410 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/363fcebf-c7da-4d18-8541-538a076f0d94-maas-api-service-ca\") pod \"363fcebf-c7da-4d18-8541-538a076f0d94\" (UID: \"363fcebf-c7da-4d18-8541-538a076f0d94\") " Aug 01 01:16:28.730836 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:28.730809 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363fcebf-c7da-4d18-8541-538a076f0d94-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "363fcebf-c7da-4d18-8541-538a076f0d94" (UID: "363fcebf-c7da-4d18-8541-538a076f0d94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:16:28.832036 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:28.831988 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/363fcebf-c7da-4d18-8541-538a076f0d94-maas-api-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:16:29.458267 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:29.458233 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" Aug 01 01:16:29.458267 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:29.458247 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j" event={"ID":"363fcebf-c7da-4d18-8541-538a076f0d94","Type":"ContainerDied","Data":"75757dd71ecfcde2ec15ba62c9676cc89f137ef18a7184ddb4c6b9801e0ca5e8"} Aug 01 01:16:29.458498 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:29.458305 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75757dd71ecfcde2ec15ba62c9676cc89f137ef18a7184ddb4c6b9801e0ca5e8" Aug 01 01:16:38.891529 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:38.891486 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:16:41.246232 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.246157 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn"] Aug 01 01:16:41.247030 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.246975 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" podUID="2387706a-2efd-42e2-90ea-2eb92bc4e07b" containerName="payload-processing" containerID="cri-o://8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c" gracePeriod=30 Aug 01 01:16:41.393882 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.393840 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j"] Aug 01 01:16:41.398074 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.398043 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ff21edf2-tbc7j"] Aug 01 01:16:41.519304 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.519276 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:16:41.544014 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.543972 2568 generic.go:356] "Generic (PLEG): container finished" podID="2387706a-2efd-42e2-90ea-2eb92bc4e07b" containerID="8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c" exitCode=0 Aug 01 01:16:41.544186 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.544020 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" Aug 01 01:16:41.544186 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.544055 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" event={"ID":"2387706a-2efd-42e2-90ea-2eb92bc4e07b","Type":"ContainerDied","Data":"8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c"} Aug 01 01:16:41.544186 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.544121 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn" event={"ID":"2387706a-2efd-42e2-90ea-2eb92bc4e07b","Type":"ContainerDied","Data":"4985ccebcff160b82aede38f2517bb23429ab86d7b456d76995ea12686f1d5eb"} Aug 01 01:16:41.544186 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.544157 2568 scope.go:117] "RemoveContainer" containerID="8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c" Aug 01 01:16:41.554923 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.554904 2568 scope.go:117] "RemoveContainer" containerID="8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c" Aug 01 01:16:41.555316 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:16:41.555284 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c\": container with ID starting with 8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c not found: ID does not exist" containerID="8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c" Aug 01 01:16:41.555408 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.555329 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c"} err="failed to get container status \"8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c\": rpc error: code = NotFound desc = could not find container \"8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c\": container with ID starting with 8f0e7e2a1b473612529b9a3a74768fa1ce4f9668957663617c9f18de95c0a75c not found: ID does not exist" Aug 01 01:16:41.662566 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.662526 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387706a-2efd-42e2-90ea-2eb92bc4e07b-plugins-config-volume\") pod \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\" (UID: \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\") " Aug 01 01:16:41.662747 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.662585 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfj2c\" (UniqueName: \"kubernetes.io/projected/2387706a-2efd-42e2-90ea-2eb92bc4e07b-kube-api-access-zfj2c\") pod \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\" (UID: \"2387706a-2efd-42e2-90ea-2eb92bc4e07b\") " Aug 01 01:16:41.662998 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.662966 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2387706a-2efd-42e2-90ea-2eb92bc4e07b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2387706a-2efd-42e2-90ea-2eb92bc4e07b" (UID: "2387706a-2efd-42e2-90ea-2eb92bc4e07b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:16:41.664821 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.664799 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2387706a-2efd-42e2-90ea-2eb92bc4e07b-kube-api-access-zfj2c" (OuterVolumeSpecName: "kube-api-access-zfj2c") pod "2387706a-2efd-42e2-90ea-2eb92bc4e07b" (UID: "2387706a-2efd-42e2-90ea-2eb92bc4e07b"). InnerVolumeSpecName "kube-api-access-zfj2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:16:41.764148 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.764049 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfj2c\" (UniqueName: \"kubernetes.io/projected/2387706a-2efd-42e2-90ea-2eb92bc4e07b-kube-api-access-zfj2c\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:16:41.764148 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.764087 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387706a-2efd-42e2-90ea-2eb92bc4e07b-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:16:41.871033 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.871001 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn"] Aug 01 01:16:41.873505 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:41.873472 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff21edf2-cf4d9f566-8q6tn"] Aug 01 01:16:42.231378 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:42.231348 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2387706a-2efd-42e2-90ea-2eb92bc4e07b" path="/var/lib/kubelet/pods/2387706a-2efd-42e2-90ea-2eb92bc4e07b/volumes" Aug 01 01:16:42.231690 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:42.231678 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="363fcebf-c7da-4d18-8541-538a076f0d94" path="/var/lib/kubelet/pods/363fcebf-c7da-4d18-8541-538a076f0d94/volumes" Aug 01 01:16:46.690230 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:16:46.690177 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:17:34.588253 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:17:34.588201 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:17:46.094431 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:17:46.094396 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:17:52.801121 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:17:52.801077 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:18:00.698070 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:00.698027 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:18:42.892303 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.892200 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k"] Aug 01 01:18:42.892731 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.892701 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2387706a-2efd-42e2-90ea-2eb92bc4e07b" containerName="payload-processing" Aug 01 01:18:42.892731 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.892718 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2387706a-2efd-42e2-90ea-2eb92bc4e07b" containerName="payload-processing" Aug 01 01:18:42.892803 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.892737 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="363fcebf-c7da-4d18-8541-538a076f0d94" containerName="revoke-keys" Aug 01 01:18:42.892803 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.892743 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="363fcebf-c7da-4d18-8541-538a076f0d94" containerName="revoke-keys" Aug 01 01:18:42.892869 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.892828 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2387706a-2efd-42e2-90ea-2eb92bc4e07b" containerName="payload-processing" Aug 01 01:18:42.892869 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.892838 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="363fcebf-c7da-4d18-8541-538a076f0d94" containerName="revoke-keys" Aug 01 01:18:42.896070 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.896052 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:42.898163 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.898139 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f6a6f5\"" Aug 01 01:18:42.898328 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.898310 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f6a6f5-dockercfg-j5s57\"" Aug 01 01:18:42.905869 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:42.905838 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k"] Aug 01 01:18:43.049239 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:43.049177 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k\" (UID: \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:43.049422 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:43.049325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7np\" (UniqueName: \"kubernetes.io/projected/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-kube-api-access-7h7np\") pod \"payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k\" (UID: \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:43.150931 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:43.150845 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7np\" (UniqueName: \"kubernetes.io/projected/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-kube-api-access-7h7np\") pod \"payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k\" (UID: \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:43.150931 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:43.150926 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k\" (UID: \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:43.151636 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:43.151615 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k\" (UID: \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:43.158945 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:43.158913 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7np\" (UniqueName: \"kubernetes.io/projected/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-kube-api-access-7h7np\") pod \"payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k\" (UID: \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:43.209586 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:43.209546 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:43.357000 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:43.356976 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k"] Aug 01 01:18:43.358994 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:18:43.358969 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf36b24c_85c8_4fc8_8c07_1ab37014aa0d.slice/crio-ffcf814f7081873a7ade76bbee71d945e0caee45e4977aec34a583ddd77d9f6c WatchSource:0}: Error finding container ffcf814f7081873a7ade76bbee71d945e0caee45e4977aec34a583ddd77d9f6c: Status 404 returned error can't find the container with id ffcf814f7081873a7ade76bbee71d945e0caee45e4977aec34a583ddd77d9f6c Aug 01 01:18:44.292957 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:44.292918 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" event={"ID":"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d","Type":"ContainerStarted","Data":"eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0"} Aug 01 01:18:44.293487 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:44.292977 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" event={"ID":"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d","Type":"ContainerStarted","Data":"ffcf814f7081873a7ade76bbee71d945e0caee45e4977aec34a583ddd77d9f6c"} Aug 01 01:18:44.293487 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:44.293058 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:18:44.307564 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:44.307507 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" podStartSLOduration=2.307488858 podStartE2EDuration="2.307488858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:18:44.3063099 +0000 UTC m=+2702.515938356" watchObservedRunningTime="2026-08-01 01:18:44.307488858 +0000 UTC m=+2702.517117347" Aug 01 01:18:44.697611 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:44.697567 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:18:55.302982 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:18:55.302947 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:19:11.923146 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:11.923100 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg"] Aug 01 01:19:11.927405 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:11.927379 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:11.929663 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:11.929638 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1cf076\"" Aug 01 01:19:11.929798 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:11.929702 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1cf076-dockercfg-z8m9f\"" Aug 01 01:19:11.941524 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:11.941489 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg"] Aug 01 01:19:12.026763 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.026725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg\" (UID: \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:12.026955 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.026780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9gn\" (UniqueName: \"kubernetes.io/projected/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-kube-api-access-sz9gn\") pod \"payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg\" (UID: \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:12.127814 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.127768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg\" (UID: \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:12.128025 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.127839 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9gn\" (UniqueName: \"kubernetes.io/projected/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-kube-api-access-sz9gn\") pod \"payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg\" (UID: \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:12.128540 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.128519 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg\" (UID: \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:12.137320 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.137289 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9gn\" (UniqueName: \"kubernetes.io/projected/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-kube-api-access-sz9gn\") pod \"payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg\" (UID: \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:12.238497 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.238465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:12.380458 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.380419 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg"] Aug 01 01:19:12.383474 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:19:12.383443 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0c46e9_847f_49a5_8505_c5a2c7e4f18d.slice/crio-d7dc7dacf81d2c0624e6e705edb12a7766be16e6160c9e344db9ac136ce2e5ba WatchSource:0}: Error finding container d7dc7dacf81d2c0624e6e705edb12a7766be16e6160c9e344db9ac136ce2e5ba: Status 404 returned error can't find the container with id d7dc7dacf81d2c0624e6e705edb12a7766be16e6160c9e344db9ac136ce2e5ba Aug 01 01:19:12.465706 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.465666 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" event={"ID":"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d","Type":"ContainerStarted","Data":"da7f1d2ab18d68c92bac55543ef651b04fcf95bf7e1ad71c9deff351dc6c0282"} Aug 01 01:19:12.465881 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.465735 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" event={"ID":"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d","Type":"ContainerStarted","Data":"d7dc7dacf81d2c0624e6e705edb12a7766be16e6160c9e344db9ac136ce2e5ba"} Aug 01 01:19:12.465881 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.465868 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:12.484963 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:12.484911 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" podStartSLOduration=1.484893457 podStartE2EDuration="1.484893457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:19:12.483520889 +0000 UTC m=+2730.693149347" watchObservedRunningTime="2026-08-01 01:19:12.484893457 +0000 UTC m=+2730.694521914" Aug 01 01:19:14.988566 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:14.988531 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:19:23.473615 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:23.473574 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" podUID="5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.69:9004: connect: connection refused" Aug 01 01:19:33.472573 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:33.472536 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:19:42.596234 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:42.596183 2568 scope.go:117] "RemoveContainer" containerID="d442ce7afecad5ed93d290a098919518326749d3fe5e9409d3ef11e0aa207f5e" Aug 01 01:19:46.695527 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:46.695493 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:19:50.102616 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:19:50.102573 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:20:21.291275 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:20:21.291237 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:20:30.891494 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:20:30.891455 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:21:11.535919 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.535885 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh"] Aug 01 01:21:11.539505 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.539474 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" Aug 01 01:21:11.541902 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.541824 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:21:11.542043 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.541875 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:21:11.550232 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.549161 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh"] Aug 01 01:21:11.632478 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.632436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88201035-c6e9-40eb-9a83-5c2e18320f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh\" (UID: \"88201035-c6e9-40eb-9a83-5c2e18320f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" Aug 01 01:21:11.733470 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.733431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88201035-c6e9-40eb-9a83-5c2e18320f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh\" (UID: \"88201035-c6e9-40eb-9a83-5c2e18320f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" Aug 01 01:21:11.734129 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.734101 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88201035-c6e9-40eb-9a83-5c2e18320f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh\" (UID: \"88201035-c6e9-40eb-9a83-5c2e18320f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" Aug 01 01:21:11.862361 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:11.862269 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" Aug 01 01:21:12.009071 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:12.009044 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh"] Aug 01 01:21:12.010746 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:21:12.010715 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88201035_c6e9_40eb_9a83_5c2e18320f9f.slice/crio-4cbd2ef81533b3e2da63d8202453d51dab1c83acce7c3735cdc769b443316c85 WatchSource:0}: Error finding container 4cbd2ef81533b3e2da63d8202453d51dab1c83acce7c3735cdc769b443316c85: Status 404 returned error can't find the container with id 4cbd2ef81533b3e2da63d8202453d51dab1c83acce7c3735cdc769b443316c85 Aug 01 01:21:12.196199 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:12.196163 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" event={"ID":"88201035-c6e9-40eb-9a83-5c2e18320f9f","Type":"ContainerStarted","Data":"4cbd2ef81533b3e2da63d8202453d51dab1c83acce7c3735cdc769b443316c85"} Aug 01 01:21:13.203676 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:13.203643 2568 generic.go:356] "Generic (PLEG): container finished" podID="88201035-c6e9-40eb-9a83-5c2e18320f9f" containerID="389ecf6375576e817a78c36ab7d96edcc7f9b3dde05dfc75dcb7c0f78958da45" exitCode=0 Aug 01 01:21:13.203676 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:13.203680 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" event={"ID":"88201035-c6e9-40eb-9a83-5c2e18320f9f","Type":"ContainerDied","Data":"389ecf6375576e817a78c36ab7d96edcc7f9b3dde05dfc75dcb7c0f78958da45"} Aug 01 01:21:14.092180 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:14.092144 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:21:14.343320 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:14.343250 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" Aug 01 01:21:14.459999 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:14.459959 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88201035-c6e9-40eb-9a83-5c2e18320f9f-maas-api-service-ca\") pod \"88201035-c6e9-40eb-9a83-5c2e18320f9f\" (UID: \"88201035-c6e9-40eb-9a83-5c2e18320f9f\") " Aug 01 01:21:14.460404 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:14.460379 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88201035-c6e9-40eb-9a83-5c2e18320f9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88201035-c6e9-40eb-9a83-5c2e18320f9f" (UID: "88201035-c6e9-40eb-9a83-5c2e18320f9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:21:14.561711 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:14.561670 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88201035-c6e9-40eb-9a83-5c2e18320f9f-maas-api-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:21:15.216255 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:15.216202 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" Aug 01 01:21:15.216456 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:15.216201 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh" event={"ID":"88201035-c6e9-40eb-9a83-5c2e18320f9f","Type":"ContainerDied","Data":"4cbd2ef81533b3e2da63d8202453d51dab1c83acce7c3735cdc769b443316c85"} Aug 01 01:21:15.216456 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:15.216346 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cbd2ef81533b3e2da63d8202453d51dab1c83acce7c3735cdc769b443316c85" Aug 01 01:21:21.939968 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:21.939915 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k"] Aug 01 01:21:21.946390 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:21.946356 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" podUID="cf36b24c-85c8-4fc8-8c07-1ab37014aa0d" containerName="payload-processing" containerID="cri-o://eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0" gracePeriod=30 Aug 01 01:21:22.118245 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.118187 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh"] Aug 01 01:21:22.121056 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.121036 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f6a6f5-rfqlh"] Aug 01 01:21:22.209076 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.209051 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:21:22.234681 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.234649 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88201035-c6e9-40eb-9a83-5c2e18320f9f" path="/var/lib/kubelet/pods/88201035-c6e9-40eb-9a83-5c2e18320f9f/volumes" Aug 01 01:21:22.239199 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.239171 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h7np\" (UniqueName: \"kubernetes.io/projected/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-kube-api-access-7h7np\") pod \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\" (UID: \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\") " Aug 01 01:21:22.240720 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.240125 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-plugins-config-volume\") pod \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\" (UID: \"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d\") " Aug 01 01:21:22.241655 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.241627 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf36b24c-85c8-4fc8-8c07-1ab37014aa0d" (UID: "cf36b24c-85c8-4fc8-8c07-1ab37014aa0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:21:22.243263 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.243188 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-kube-api-access-7h7np" (OuterVolumeSpecName: "kube-api-access-7h7np") pod "cf36b24c-85c8-4fc8-8c07-1ab37014aa0d" (UID: "cf36b24c-85c8-4fc8-8c07-1ab37014aa0d"). InnerVolumeSpecName "kube-api-access-7h7np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:21:22.267861 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.267823 2568 generic.go:356] "Generic (PLEG): container finished" podID="cf36b24c-85c8-4fc8-8c07-1ab37014aa0d" containerID="eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0" exitCode=0 Aug 01 01:21:22.268016 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.267880 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" Aug 01 01:21:22.268016 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.267903 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" event={"ID":"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d","Type":"ContainerDied","Data":"eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0"} Aug 01 01:21:22.268016 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.267968 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k" event={"ID":"cf36b24c-85c8-4fc8-8c07-1ab37014aa0d","Type":"ContainerDied","Data":"ffcf814f7081873a7ade76bbee71d945e0caee45e4977aec34a583ddd77d9f6c"} Aug 01 01:21:22.268016 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.267989 2568 scope.go:117] "RemoveContainer" containerID="eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0" Aug 01 01:21:22.277744 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.277728 2568 scope.go:117] "RemoveContainer" containerID="eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0" Aug 01 01:21:22.277993 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:21:22.277975 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0\": container with ID starting with eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0 not found: ID does not exist" containerID="eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0" Aug 01 01:21:22.278048 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.278002 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0"} err="failed to get container status \"eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0\": rpc error: code = NotFound desc = could not find container \"eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0\": container with ID starting with eb292a46d61681c430ae86a2bacd59ede7ee5f86f7a4efc487ce4ad0c05774a0 not found: ID does not exist" Aug 01 01:21:22.289804 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.289773 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k"] Aug 01 01:21:22.293122 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.293099 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f6a6f5-9b798f4-6569k"] Aug 01 01:21:22.341459 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.341417 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7np\" (UniqueName: \"kubernetes.io/projected/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-kube-api-access-7h7np\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:21:22.341459 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:22.341450 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:21:23.992554 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:23.992519 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:21:24.231492 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:24.231463 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf36b24c-85c8-4fc8-8c07-1ab37014aa0d" path="/var/lib/kubelet/pods/cf36b24c-85c8-4fc8-8c07-1ab37014aa0d/volumes" Aug 01 01:21:28.820187 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.820151 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w"] Aug 01 01:21:28.820640 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.820626 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88201035-c6e9-40eb-9a83-5c2e18320f9f" containerName="revoke-keys" Aug 01 01:21:28.820684 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.820642 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="88201035-c6e9-40eb-9a83-5c2e18320f9f" containerName="revoke-keys" Aug 01 01:21:28.820684 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.820650 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf36b24c-85c8-4fc8-8c07-1ab37014aa0d" containerName="payload-processing" Aug 01 01:21:28.820684 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.820656 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf36b24c-85c8-4fc8-8c07-1ab37014aa0d" containerName="payload-processing" Aug 01 01:21:28.820774 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.820733 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf36b24c-85c8-4fc8-8c07-1ab37014aa0d" containerName="payload-processing" Aug 01 01:21:28.820774 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.820743 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="88201035-c6e9-40eb-9a83-5c2e18320f9f" containerName="revoke-keys" Aug 01 01:21:28.825131 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.825114 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" Aug 01 01:21:28.827625 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.827599 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:21:28.827740 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.827599 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:21:28.830361 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.830337 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w"] Aug 01 01:21:28.904457 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:28.904418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d745e66f-8d0f-4fff-b20b-df8c2c67567b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w\" (UID: \"d745e66f-8d0f-4fff-b20b-df8c2c67567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" Aug 01 01:21:29.005705 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:29.005664 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d745e66f-8d0f-4fff-b20b-df8c2c67567b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w\" (UID: \"d745e66f-8d0f-4fff-b20b-df8c2c67567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" Aug 01 01:21:29.006431 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:29.006406 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d745e66f-8d0f-4fff-b20b-df8c2c67567b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w\" (UID: \"d745e66f-8d0f-4fff-b20b-df8c2c67567b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" Aug 01 01:21:29.137778 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:29.137692 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" Aug 01 01:21:29.280318 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:29.280285 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w"] Aug 01 01:21:29.281103 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:21:29.281067 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd745e66f_8d0f_4fff_b20b_df8c2c67567b.slice/crio-80bb4fda9c0a8721edcf236b9e2da4aee4633aaec2356937a0c0b13e0248a95a WatchSource:0}: Error finding container 80bb4fda9c0a8721edcf236b9e2da4aee4633aaec2356937a0c0b13e0248a95a: Status 404 returned error can't find the container with id 80bb4fda9c0a8721edcf236b9e2da4aee4633aaec2356937a0c0b13e0248a95a Aug 01 01:21:29.312275 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:29.312244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" event={"ID":"d745e66f-8d0f-4fff-b20b-df8c2c67567b","Type":"ContainerStarted","Data":"80bb4fda9c0a8721edcf236b9e2da4aee4633aaec2356937a0c0b13e0248a95a"} Aug 01 01:21:30.320842 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:30.320759 2568 generic.go:356] "Generic (PLEG): container finished" podID="d745e66f-8d0f-4fff-b20b-df8c2c67567b" containerID="f82816dba96ed3e901c3a2cb18c629f767854a8c15ca361747f88a1621d637a7" exitCode=0 Aug 01 01:21:30.320842 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:30.320831 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" event={"ID":"d745e66f-8d0f-4fff-b20b-df8c2c67567b","Type":"ContainerDied","Data":"f82816dba96ed3e901c3a2cb18c629f767854a8c15ca361747f88a1621d637a7"} Aug 01 01:21:31.470677 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:31.470648 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" Aug 01 01:21:31.530904 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:31.530873 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d745e66f-8d0f-4fff-b20b-df8c2c67567b-maas-api-service-ca\") pod \"d745e66f-8d0f-4fff-b20b-df8c2c67567b\" (UID: \"d745e66f-8d0f-4fff-b20b-df8c2c67567b\") " Aug 01 01:21:31.531249 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:31.531201 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d745e66f-8d0f-4fff-b20b-df8c2c67567b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d745e66f-8d0f-4fff-b20b-df8c2c67567b" (UID: "d745e66f-8d0f-4fff-b20b-df8c2c67567b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:21:31.631664 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:31.631579 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d745e66f-8d0f-4fff-b20b-df8c2c67567b-maas-api-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:21:32.333972 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:32.333937 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" Aug 01 01:21:32.333972 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:32.333967 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w" event={"ID":"d745e66f-8d0f-4fff-b20b-df8c2c67567b","Type":"ContainerDied","Data":"80bb4fda9c0a8721edcf236b9e2da4aee4633aaec2356937a0c0b13e0248a95a"} Aug 01 01:21:32.334189 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:32.334015 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80bb4fda9c0a8721edcf236b9e2da4aee4633aaec2356937a0c0b13e0248a95a" Aug 01 01:21:39.224423 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.224343 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg"] Aug 01 01:21:39.224876 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.224818 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" podUID="5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" containerName="payload-pre-processing" containerID="cri-o://da7f1d2ab18d68c92bac55543ef651b04fcf95bf7e1ad71c9deff351dc6c0282" gracePeriod=30 Aug 01 01:21:39.344783 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.344759 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w"] Aug 01 01:21:39.347244 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.347202 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1cf076-mll2w"] Aug 01 01:21:39.380615 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.380582 2568 generic.go:356] "Generic (PLEG): container finished" podID="5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" containerID="da7f1d2ab18d68c92bac55543ef651b04fcf95bf7e1ad71c9deff351dc6c0282" exitCode=0 Aug 01 01:21:39.380808 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.380663 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" event={"ID":"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d","Type":"ContainerDied","Data":"da7f1d2ab18d68c92bac55543ef651b04fcf95bf7e1ad71c9deff351dc6c0282"} Aug 01 01:21:39.506949 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.506926 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:21:39.610750 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.610724 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-plugins-config-volume\") pod \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\" (UID: \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\") " Aug 01 01:21:39.610927 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.610783 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz9gn\" (UniqueName: \"kubernetes.io/projected/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-kube-api-access-sz9gn\") pod \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\" (UID: \"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d\") " Aug 01 01:21:39.611146 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.611125 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" (UID: "5a0c46e9-847f-49a5-8505-c5a2c7e4f18d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:21:39.613009 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.612990 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-kube-api-access-sz9gn" (OuterVolumeSpecName: "kube-api-access-sz9gn") pod "5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" (UID: "5a0c46e9-847f-49a5-8505-c5a2c7e4f18d"). InnerVolumeSpecName "kube-api-access-sz9gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:21:39.712135 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.712096 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:21:39.712135 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:39.712129 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9gn\" (UniqueName: \"kubernetes.io/projected/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d-kube-api-access-sz9gn\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:21:40.230956 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:40.230927 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d745e66f-8d0f-4fff-b20b-df8c2c67567b" path="/var/lib/kubelet/pods/d745e66f-8d0f-4fff-b20b-df8c2c67567b/volumes" Aug 01 01:21:40.388188 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:40.388161 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" Aug 01 01:21:40.388391 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:40.388189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg" event={"ID":"5a0c46e9-847f-49a5-8505-c5a2c7e4f18d","Type":"ContainerDied","Data":"d7dc7dacf81d2c0624e6e705edb12a7766be16e6160c9e344db9ac136ce2e5ba"} Aug 01 01:21:40.388391 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:40.388275 2568 scope.go:117] "RemoveContainer" containerID="da7f1d2ab18d68c92bac55543ef651b04fcf95bf7e1ad71c9deff351dc6c0282" Aug 01 01:21:40.404259 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:40.404230 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg"] Aug 01 01:21:40.407128 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:40.407101 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1cf076-59f779c8cb-w6tzg"] Aug 01 01:21:41.386318 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:41.386286 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:21:42.232062 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:42.232028 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" path="/var/lib/kubelet/pods/5a0c46e9-847f-49a5-8505-c5a2c7e4f18d/volumes" Aug 01 01:21:50.755115 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.755075 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw"] Aug 01 01:21:50.755512 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.755499 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d745e66f-8d0f-4fff-b20b-df8c2c67567b" containerName="revoke-keys" Aug 01 01:21:50.755551 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.755513 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d745e66f-8d0f-4fff-b20b-df8c2c67567b" containerName="revoke-keys" Aug 01 01:21:50.755551 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.755540 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" containerName="payload-pre-processing" Aug 01 01:21:50.755551 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.755546 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" containerName="payload-pre-processing" Aug 01 01:21:50.755655 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.755602 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0c46e9-847f-49a5-8505-c5a2c7e4f18d" containerName="payload-pre-processing" Aug 01 01:21:50.755655 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.755613 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d745e66f-8d0f-4fff-b20b-df8c2c67567b" containerName="revoke-keys" Aug 01 01:21:50.758862 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.758846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:50.761172 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.761150 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-041b9f-dockercfg-4948w\"" Aug 01 01:21:50.761331 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.761175 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-041b9f\"" Aug 01 01:21:50.768616 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.768593 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw"] Aug 01 01:21:50.922238 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.922170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqfs2\" (UniqueName: \"kubernetes.io/projected/051b91e2-af40-44d2-816c-e2755b661a91-kube-api-access-qqfs2\") pod \"payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw\" (UID: \"051b91e2-af40-44d2-816c-e2755b661a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:50.922424 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:50.922289 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b91e2-af40-44d2-816c-e2755b661a91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw\" (UID: \"051b91e2-af40-44d2-816c-e2755b661a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:51.023388 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.023304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfs2\" (UniqueName: \"kubernetes.io/projected/051b91e2-af40-44d2-816c-e2755b661a91-kube-api-access-qqfs2\") pod \"payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw\" (UID: \"051b91e2-af40-44d2-816c-e2755b661a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:51.023538 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.023391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b91e2-af40-44d2-816c-e2755b661a91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw\" (UID: \"051b91e2-af40-44d2-816c-e2755b661a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:51.024067 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.024045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b91e2-af40-44d2-816c-e2755b661a91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw\" (UID: \"051b91e2-af40-44d2-816c-e2755b661a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:51.034628 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.034600 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfs2\" (UniqueName: \"kubernetes.io/projected/051b91e2-af40-44d2-816c-e2755b661a91-kube-api-access-qqfs2\") pod \"payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw\" (UID: \"051b91e2-af40-44d2-816c-e2755b661a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:51.070778 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.070738 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:51.211557 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.211527 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw"] Aug 01 01:21:51.212681 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:21:51.212656 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051b91e2_af40_44d2_816c_e2755b661a91.slice/crio-d5380d8c04d1f097c7e0c3a59dd8de777acfe64a721935272b4ce82d87fb876c WatchSource:0}: Error finding container d5380d8c04d1f097c7e0c3a59dd8de777acfe64a721935272b4ce82d87fb876c: Status 404 returned error can't find the container with id d5380d8c04d1f097c7e0c3a59dd8de777acfe64a721935272b4ce82d87fb876c Aug 01 01:21:51.457667 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.457629 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" event={"ID":"051b91e2-af40-44d2-816c-e2755b661a91","Type":"ContainerStarted","Data":"6d2445f0a824e3491180f5c8292edc28902f36c50e1306d74d4554a6eaa8bd5e"} Aug 01 01:21:51.457867 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.457680 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" event={"ID":"051b91e2-af40-44d2-816c-e2755b661a91","Type":"ContainerStarted","Data":"d5380d8c04d1f097c7e0c3a59dd8de777acfe64a721935272b4ce82d87fb876c"} Aug 01 01:21:51.457867 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.457846 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:21:51.474416 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:51.474367 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" podStartSLOduration=1.474353553 podStartE2EDuration="1.474353553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:21:51.471714313 +0000 UTC m=+2889.681342771" watchObservedRunningTime="2026-08-01 01:21:51.474353553 +0000 UTC m=+2889.683982008" Aug 01 01:21:53.199568 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:21:53.199537 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:22:02.466375 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:02.466347 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:22:20.193003 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.192960 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk"] Aug 01 01:22:20.198022 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.197999 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:20.200484 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.200460 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a9f952\"" Aug 01 01:22:20.200625 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.200460 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a9f952-dockercfg-6vkq2\"" Aug 01 01:22:20.205566 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.205539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk"] Aug 01 01:22:20.298872 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.298823 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760746f3-b91c-4fd2-aa3b-8769c72914c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk\" (UID: \"760746f3-b91c-4fd2-aa3b-8769c72914c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:20.299074 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.298884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2vd\" (UniqueName: \"kubernetes.io/projected/760746f3-b91c-4fd2-aa3b-8769c72914c0-kube-api-access-bw2vd\") pod \"payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk\" (UID: \"760746f3-b91c-4fd2-aa3b-8769c72914c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:20.400279 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.400245 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760746f3-b91c-4fd2-aa3b-8769c72914c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk\" (UID: \"760746f3-b91c-4fd2-aa3b-8769c72914c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:20.400434 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.400309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2vd\" (UniqueName: \"kubernetes.io/projected/760746f3-b91c-4fd2-aa3b-8769c72914c0-kube-api-access-bw2vd\") pod \"payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk\" (UID: \"760746f3-b91c-4fd2-aa3b-8769c72914c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:20.400868 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.400847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760746f3-b91c-4fd2-aa3b-8769c72914c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk\" (UID: \"760746f3-b91c-4fd2-aa3b-8769c72914c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:20.407365 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.407345 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2vd\" (UniqueName: \"kubernetes.io/projected/760746f3-b91c-4fd2-aa3b-8769c72914c0-kube-api-access-bw2vd\") pod \"payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk\" (UID: \"760746f3-b91c-4fd2-aa3b-8769c72914c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:20.511255 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.511193 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:20.656658 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:20.656632 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk"] Aug 01 01:22:20.659137 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:22:20.659100 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760746f3_b91c_4fd2_aa3b_8769c72914c0.slice/crio-599d2b2d9b4e3e1fee7cae8f21dee2767373719123fc6390ab2512271595b637 WatchSource:0}: Error finding container 599d2b2d9b4e3e1fee7cae8f21dee2767373719123fc6390ab2512271595b637: Status 404 returned error can't find the container with id 599d2b2d9b4e3e1fee7cae8f21dee2767373719123fc6390ab2512271595b637 Aug 01 01:22:21.649891 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:21.649857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" event={"ID":"760746f3-b91c-4fd2-aa3b-8769c72914c0","Type":"ContainerStarted","Data":"c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe"} Aug 01 01:22:21.650329 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:21.649909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" event={"ID":"760746f3-b91c-4fd2-aa3b-8769c72914c0","Type":"ContainerStarted","Data":"599d2b2d9b4e3e1fee7cae8f21dee2767373719123fc6390ab2512271595b637"} Aug 01 01:22:21.650329 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:21.650044 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:21.664688 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:21.664646 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" podStartSLOduration=1.6646314420000001 podStartE2EDuration="1.664631442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:22:21.663334771 +0000 UTC m=+2919.872963226" watchObservedRunningTime="2026-08-01 01:22:21.664631442 +0000 UTC m=+2919.874259960" Aug 01 01:22:21.900807 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:21.900723 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:22:32.659397 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:32.659365 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:22:42.646385 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:42.646352 2568 scope.go:117] "RemoveContainer" containerID="3a329aef8fb0adc0d78a62d554c1065a89e2072e99c9dc26a6c87725a9e6f33e" Aug 01 01:22:46.292652 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.292621 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t"] Aug 01 01:22:46.297700 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.297682 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" Aug 01 01:22:46.300858 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.300834 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:22:46.301106 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.301089 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:22:46.310509 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.310486 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t"] Aug 01 01:22:46.359528 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.359489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3862dca8-36ed-42c3-b6ac-028b2e69e9f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-041b9f-4s82t\" (UID: \"3862dca8-36ed-42c3-b6ac-028b2e69e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" Aug 01 01:22:46.460565 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.460524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3862dca8-36ed-42c3-b6ac-028b2e69e9f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-041b9f-4s82t\" (UID: \"3862dca8-36ed-42c3-b6ac-028b2e69e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" Aug 01 01:22:46.461167 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.461144 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3862dca8-36ed-42c3-b6ac-028b2e69e9f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-041b9f-4s82t\" (UID: \"3862dca8-36ed-42c3-b6ac-028b2e69e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" Aug 01 01:22:46.608161 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.608058 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" Aug 01 01:22:46.743415 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.743380 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t"] Aug 01 01:22:46.745986 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:22:46.745955 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3862dca8_36ed_42c3_b6ac_028b2e69e9f2.slice/crio-0ef593f541d3b4b739ebf61bf5bd213354f2df10b0b2a235bffe867a800fc50c WatchSource:0}: Error finding container 0ef593f541d3b4b739ebf61bf5bd213354f2df10b0b2a235bffe867a800fc50c: Status 404 returned error can't find the container with id 0ef593f541d3b4b739ebf61bf5bd213354f2df10b0b2a235bffe867a800fc50c Aug 01 01:22:46.808945 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:46.808908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" event={"ID":"3862dca8-36ed-42c3-b6ac-028b2e69e9f2","Type":"ContainerStarted","Data":"0ef593f541d3b4b739ebf61bf5bd213354f2df10b0b2a235bffe867a800fc50c"} Aug 01 01:22:47.817267 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:47.817157 2568 generic.go:356] "Generic (PLEG): container finished" podID="3862dca8-36ed-42c3-b6ac-028b2e69e9f2" containerID="7964938b386a6c17a8ee510fcfd12ce09bf2bd1aed515a596b99fc9e5dfcbbdd" exitCode=0 Aug 01 01:22:47.817267 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:47.817244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" event={"ID":"3862dca8-36ed-42c3-b6ac-028b2e69e9f2","Type":"ContainerDied","Data":"7964938b386a6c17a8ee510fcfd12ce09bf2bd1aed515a596b99fc9e5dfcbbdd"} Aug 01 01:22:48.960261 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:48.960234 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" Aug 01 01:22:48.987830 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:48.987801 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3862dca8-36ed-42c3-b6ac-028b2e69e9f2-maas-api-service-ca\") pod \"3862dca8-36ed-42c3-b6ac-028b2e69e9f2\" (UID: \"3862dca8-36ed-42c3-b6ac-028b2e69e9f2\") " Aug 01 01:22:48.988122 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:48.988103 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3862dca8-36ed-42c3-b6ac-028b2e69e9f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3862dca8-36ed-42c3-b6ac-028b2e69e9f2" (UID: "3862dca8-36ed-42c3-b6ac-028b2e69e9f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:22:48.988231 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:48.988203 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3862dca8-36ed-42c3-b6ac-028b2e69e9f2-maas-api-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:22:49.831116 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:49.831084 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" Aug 01 01:22:49.831312 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:49.831115 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t" event={"ID":"3862dca8-36ed-42c3-b6ac-028b2e69e9f2","Type":"ContainerDied","Data":"0ef593f541d3b4b739ebf61bf5bd213354f2df10b0b2a235bffe867a800fc50c"} Aug 01 01:22:49.831312 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:22:49.831165 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef593f541d3b4b739ebf61bf5bd213354f2df10b0b2a235bffe867a800fc50c" Aug 01 01:23:01.719553 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:01.719514 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw"] Aug 01 01:23:01.720072 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:01.719771 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" podUID="051b91e2-af40-44d2-816c-e2755b661a91" containerName="payload-processing" containerID="cri-o://6d2445f0a824e3491180f5c8292edc28902f36c50e1306d74d4554a6eaa8bd5e" gracePeriod=30 Aug 01 01:23:01.841591 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:01.841565 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t"] Aug 01 01:23:01.847121 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:01.847091 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-041b9f-4s82t"] Aug 01 01:23:01.908698 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:01.908660 2568 generic.go:356] "Generic (PLEG): container finished" podID="051b91e2-af40-44d2-816c-e2755b661a91" containerID="6d2445f0a824e3491180f5c8292edc28902f36c50e1306d74d4554a6eaa8bd5e" exitCode=0 Aug 01 01:23:01.908863 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:01.908703 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" event={"ID":"051b91e2-af40-44d2-816c-e2755b661a91","Type":"ContainerDied","Data":"6d2445f0a824e3491180f5c8292edc28902f36c50e1306d74d4554a6eaa8bd5e"} Aug 01 01:23:01.983655 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:01.983580 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:23:02.123439 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.123406 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqfs2\" (UniqueName: \"kubernetes.io/projected/051b91e2-af40-44d2-816c-e2755b661a91-kube-api-access-qqfs2\") pod \"051b91e2-af40-44d2-816c-e2755b661a91\" (UID: \"051b91e2-af40-44d2-816c-e2755b661a91\") " Aug 01 01:23:02.123642 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.123457 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b91e2-af40-44d2-816c-e2755b661a91-plugins-config-volume\") pod \"051b91e2-af40-44d2-816c-e2755b661a91\" (UID: \"051b91e2-af40-44d2-816c-e2755b661a91\") " Aug 01 01:23:02.123883 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.123863 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051b91e2-af40-44d2-816c-e2755b661a91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "051b91e2-af40-44d2-816c-e2755b661a91" (UID: "051b91e2-af40-44d2-816c-e2755b661a91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:23:02.125749 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.125732 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051b91e2-af40-44d2-816c-e2755b661a91-kube-api-access-qqfs2" (OuterVolumeSpecName: "kube-api-access-qqfs2") pod "051b91e2-af40-44d2-816c-e2755b661a91" (UID: "051b91e2-af40-44d2-816c-e2755b661a91"). InnerVolumeSpecName "kube-api-access-qqfs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:23:02.225125 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.225085 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqfs2\" (UniqueName: \"kubernetes.io/projected/051b91e2-af40-44d2-816c-e2755b661a91-kube-api-access-qqfs2\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:23:02.225125 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.225117 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051b91e2-af40-44d2-816c-e2755b661a91-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:23:02.231362 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.231333 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3862dca8-36ed-42c3-b6ac-028b2e69e9f2" path="/var/lib/kubelet/pods/3862dca8-36ed-42c3-b6ac-028b2e69e9f2/volumes" Aug 01 01:23:02.916257 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.916165 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" event={"ID":"051b91e2-af40-44d2-816c-e2755b661a91","Type":"ContainerDied","Data":"d5380d8c04d1f097c7e0c3a59dd8de777acfe64a721935272b4ce82d87fb876c"} Aug 01 01:23:02.916670 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.916183 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw" Aug 01 01:23:02.916670 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.916304 2568 scope.go:117] "RemoveContainer" containerID="6d2445f0a824e3491180f5c8292edc28902f36c50e1306d74d4554a6eaa8bd5e" Aug 01 01:23:02.931725 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.931682 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw"] Aug 01 01:23:02.934998 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:02.934974 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-041b9f-58967b45fb-p5ffw"] Aug 01 01:23:04.099131 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:04.099097 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:23:04.231195 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:04.231158 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="051b91e2-af40-44d2-816c-e2755b661a91" path="/var/lib/kubelet/pods/051b91e2-af40-44d2-816c-e2755b661a91/volumes" Aug 01 01:23:08.553852 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.553811 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9"] Aug 01 01:23:08.554272 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.554259 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3862dca8-36ed-42c3-b6ac-028b2e69e9f2" containerName="revoke-keys" Aug 01 01:23:08.554272 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.554273 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3862dca8-36ed-42c3-b6ac-028b2e69e9f2" containerName="revoke-keys" Aug 01 01:23:08.554381 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.554283 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="051b91e2-af40-44d2-816c-e2755b661a91" containerName="payload-processing" Aug 01 01:23:08.554381 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.554289 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="051b91e2-af40-44d2-816c-e2755b661a91" containerName="payload-processing" Aug 01 01:23:08.554381 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.554365 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="051b91e2-af40-44d2-816c-e2755b661a91" containerName="payload-processing" Aug 01 01:23:08.554381 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.554377 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3862dca8-36ed-42c3-b6ac-028b2e69e9f2" containerName="revoke-keys" Aug 01 01:23:08.558896 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.558876 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" Aug 01 01:23:08.560959 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.560936 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:23:08.561102 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.560942 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:23:08.563915 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.563888 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9"] Aug 01 01:23:08.693886 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.693849 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd94092-1005-49cb-a2fb-4b3ea5e00da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9\" (UID: \"5bd94092-1005-49cb-a2fb-4b3ea5e00da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" Aug 01 01:23:08.795154 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.795124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd94092-1005-49cb-a2fb-4b3ea5e00da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9\" (UID: \"5bd94092-1005-49cb-a2fb-4b3ea5e00da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" Aug 01 01:23:08.795780 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.795760 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd94092-1005-49cb-a2fb-4b3ea5e00da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9\" (UID: \"5bd94092-1005-49cb-a2fb-4b3ea5e00da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" Aug 01 01:23:08.872078 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:08.871997 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" Aug 01 01:23:09.029819 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:09.029795 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9"] Aug 01 01:23:09.031601 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:23:09.031573 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd94092_1005_49cb_a2fb_4b3ea5e00da8.slice/crio-13c4dd4d8c7e71389b27db8d86da32c51295bffcf873509c92fa5276e92d92d4 WatchSource:0}: Error finding container 13c4dd4d8c7e71389b27db8d86da32c51295bffcf873509c92fa5276e92d92d4: Status 404 returned error can't find the container with id 13c4dd4d8c7e71389b27db8d86da32c51295bffcf873509c92fa5276e92d92d4 Aug 01 01:23:09.962977 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:09.962943 2568 generic.go:356] "Generic (PLEG): container finished" podID="5bd94092-1005-49cb-a2fb-4b3ea5e00da8" containerID="bacbc5619ab3c6ce97f4f9e32f5dca85268d299b657b343fbd61159dd2c3bc8b" exitCode=0 Aug 01 01:23:09.963431 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:09.962980 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" event={"ID":"5bd94092-1005-49cb-a2fb-4b3ea5e00da8","Type":"ContainerDied","Data":"bacbc5619ab3c6ce97f4f9e32f5dca85268d299b657b343fbd61159dd2c3bc8b"} Aug 01 01:23:09.963431 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:09.963037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" event={"ID":"5bd94092-1005-49cb-a2fb-4b3ea5e00da8","Type":"ContainerStarted","Data":"13c4dd4d8c7e71389b27db8d86da32c51295bffcf873509c92fa5276e92d92d4"} Aug 01 01:23:11.110641 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:11.110618 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" Aug 01 01:23:11.217853 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:11.217818 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd94092-1005-49cb-a2fb-4b3ea5e00da8-maas-api-service-ca\") pod \"5bd94092-1005-49cb-a2fb-4b3ea5e00da8\" (UID: \"5bd94092-1005-49cb-a2fb-4b3ea5e00da8\") " Aug 01 01:23:11.218205 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:11.218183 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd94092-1005-49cb-a2fb-4b3ea5e00da8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bd94092-1005-49cb-a2fb-4b3ea5e00da8" (UID: "5bd94092-1005-49cb-a2fb-4b3ea5e00da8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:23:11.319499 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:11.319406 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd94092-1005-49cb-a2fb-4b3ea5e00da8-maas-api-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:23:11.975695 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:11.975669 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" Aug 01 01:23:11.975870 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:11.975710 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9" event={"ID":"5bd94092-1005-49cb-a2fb-4b3ea5e00da8","Type":"ContainerDied","Data":"13c4dd4d8c7e71389b27db8d86da32c51295bffcf873509c92fa5276e92d92d4"} Aug 01 01:23:11.975870 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:11.975757 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13c4dd4d8c7e71389b27db8d86da32c51295bffcf873509c92fa5276e92d92d4" Aug 01 01:23:24.157899 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.157861 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk"] Aug 01 01:23:24.160457 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.158145 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" podUID="760746f3-b91c-4fd2-aa3b-8769c72914c0" containerName="payload-pre-processing" containerID="cri-o://c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe" gracePeriod=30 Aug 01 01:23:24.289372 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.289343 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9"] Aug 01 01:23:24.291857 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.291826 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9f952-ck4d9"] Aug 01 01:23:24.431725 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.431695 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:23:24.549186 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.549154 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760746f3-b91c-4fd2-aa3b-8769c72914c0-plugins-config-volume\") pod \"760746f3-b91c-4fd2-aa3b-8769c72914c0\" (UID: \"760746f3-b91c-4fd2-aa3b-8769c72914c0\") " Aug 01 01:23:24.549403 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.549293 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw2vd\" (UniqueName: \"kubernetes.io/projected/760746f3-b91c-4fd2-aa3b-8769c72914c0-kube-api-access-bw2vd\") pod \"760746f3-b91c-4fd2-aa3b-8769c72914c0\" (UID: \"760746f3-b91c-4fd2-aa3b-8769c72914c0\") " Aug 01 01:23:24.549551 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.549527 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760746f3-b91c-4fd2-aa3b-8769c72914c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "760746f3-b91c-4fd2-aa3b-8769c72914c0" (UID: "760746f3-b91c-4fd2-aa3b-8769c72914c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:23:24.549630 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.549610 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760746f3-b91c-4fd2-aa3b-8769c72914c0-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:23:24.551575 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.551551 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760746f3-b91c-4fd2-aa3b-8769c72914c0-kube-api-access-bw2vd" (OuterVolumeSpecName: "kube-api-access-bw2vd") pod "760746f3-b91c-4fd2-aa3b-8769c72914c0" (UID: "760746f3-b91c-4fd2-aa3b-8769c72914c0"). InnerVolumeSpecName "kube-api-access-bw2vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:23:24.650979 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:24.650942 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw2vd\" (UniqueName: \"kubernetes.io/projected/760746f3-b91c-4fd2-aa3b-8769c72914c0-kube-api-access-bw2vd\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:23:25.054723 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.054692 2568 generic.go:356] "Generic (PLEG): container finished" podID="760746f3-b91c-4fd2-aa3b-8769c72914c0" containerID="c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe" exitCode=0 Aug 01 01:23:25.054912 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.054744 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" Aug 01 01:23:25.054912 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.054789 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" event={"ID":"760746f3-b91c-4fd2-aa3b-8769c72914c0","Type":"ContainerDied","Data":"c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe"} Aug 01 01:23:25.054912 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.054852 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk" event={"ID":"760746f3-b91c-4fd2-aa3b-8769c72914c0","Type":"ContainerDied","Data":"599d2b2d9b4e3e1fee7cae8f21dee2767373719123fc6390ab2512271595b637"} Aug 01 01:23:25.054912 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.054882 2568 scope.go:117] "RemoveContainer" containerID="c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe" Aug 01 01:23:25.065797 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.065775 2568 scope.go:117] "RemoveContainer" containerID="c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe" Aug 01 01:23:25.066098 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:23:25.066073 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe\": container with ID starting with c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe not found: ID does not exist" containerID="c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe" Aug 01 01:23:25.066161 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.066109 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe"} err="failed to get container status \"c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe\": rpc error: code = NotFound desc = could not find container \"c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe\": container with ID starting with c2224c0f9698beb8ebbdf79d2e52127c16eeeae46afa7349dee3ddca9467befe not found: ID does not exist" Aug 01 01:23:25.081092 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.081052 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk"] Aug 01 01:23:25.084554 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:25.084531 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9f952-786c8b4644-7k2hk"] Aug 01 01:23:26.230623 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:26.230586 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd94092-1005-49cb-a2fb-4b3ea5e00da8" path="/var/lib/kubelet/pods/5bd94092-1005-49cb-a2fb-4b3ea5e00da8/volumes" Aug 01 01:23:26.230995 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:26.230959 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760746f3-b91c-4fd2-aa3b-8769c72914c0" path="/var/lib/kubelet/pods/760746f3-b91c-4fd2-aa3b-8769c72914c0/volumes" Aug 01 01:23:26.494980 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:26.494894 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:23:35.659865 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.659819 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz"] Aug 01 01:23:35.660513 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.660495 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760746f3-b91c-4fd2-aa3b-8769c72914c0" containerName="payload-pre-processing" Aug 01 01:23:35.660581 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.660515 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="760746f3-b91c-4fd2-aa3b-8769c72914c0" containerName="payload-pre-processing" Aug 01 01:23:35.660581 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.660541 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd94092-1005-49cb-a2fb-4b3ea5e00da8" containerName="revoke-keys" Aug 01 01:23:35.660581 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.660549 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd94092-1005-49cb-a2fb-4b3ea5e00da8" containerName="revoke-keys" Aug 01 01:23:35.660733 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.660642 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="760746f3-b91c-4fd2-aa3b-8769c72914c0" containerName="payload-pre-processing" Aug 01 01:23:35.660733 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.660655 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd94092-1005-49cb-a2fb-4b3ea5e00da8" containerName="revoke-keys" Aug 01 01:23:35.665512 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.665489 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.667790 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.667683 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-423d15-serving-cert\"" Aug 01 01:23:35.668176 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.668010 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 01 01:23:35.668176 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.668079 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c4l5w\"" Aug 01 01:23:35.668346 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.668295 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:23:35.671766 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.671745 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz"] Aug 01 01:23:35.764658 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.764591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvj2\" (UniqueName: \"kubernetes.io/projected/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-kube-api-access-qrvj2\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.764658 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.764657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-openshift-service-ca\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.764909 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.764720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-maas-api-tls\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.865537 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.865497 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvj2\" (UniqueName: \"kubernetes.io/projected/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-kube-api-access-qrvj2\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.865726 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.865557 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-openshift-service-ca\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.865726 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.865611 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-maas-api-tls\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.866307 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.866279 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-openshift-service-ca\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.868125 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.868107 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-maas-api-tls\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.874044 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.874021 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvj2\" (UniqueName: \"kubernetes.io/projected/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-kube-api-access-qrvj2\") pod \"maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:35.978864 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:35.978826 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:36.122726 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:36.122693 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz"] Aug 01 01:23:36.124152 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:23:36.124115 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2506d8a_2de5_4b54_a1b1_cce4d01adb75.slice/crio-f7c0500f5858e6e28710508a1ed31a0847d2df0b9660ce06f211eb3f4c80d5ef WatchSource:0}: Error finding container f7c0500f5858e6e28710508a1ed31a0847d2df0b9660ce06f211eb3f4c80d5ef: Status 404 returned error can't find the container with id f7c0500f5858e6e28710508a1ed31a0847d2df0b9660ce06f211eb3f4c80d5ef Aug 01 01:23:36.125419 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:36.125400 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 01:23:37.144889 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:37.144851 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" event={"ID":"c2506d8a-2de5-4b54-a1b1-cce4d01adb75","Type":"ContainerStarted","Data":"f7c0500f5858e6e28710508a1ed31a0847d2df0b9660ce06f211eb3f4c80d5ef"} Aug 01 01:23:38.153326 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:38.153234 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" event={"ID":"c2506d8a-2de5-4b54-a1b1-cce4d01adb75","Type":"ContainerStarted","Data":"d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550"} Aug 01 01:23:38.153326 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:38.153293 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:23:38.169227 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:38.169156 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" podStartSLOduration=1.571818334 podStartE2EDuration="3.169139984s" totalImagesPullingTime="1.59732165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:23:38.16736516 +0000 UTC m=+2996.376993629" watchObservedRunningTime="2026-08-01 01:23:38.169139984 +0000 UTC m=+2996.378768439" Aug 01 01:23:39.486634 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:39.486598 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:23:44.164614 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:23:44.164581 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:24:05.417281 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.417238 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr"] Aug 01 01:24:05.423026 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.422903 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:05.426343 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.426315 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-65669d\"" Aug 01 01:24:05.426525 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.426317 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-65669d-dockercfg-8c265\"" Aug 01 01:24:05.428781 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.428756 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr"] Aug 01 01:24:05.443816 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.443790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4543d7-fc65-4c3c-8326-56fecd858313-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr\" (UID: \"2d4543d7-fc65-4c3c-8326-56fecd858313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:05.443967 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.443868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxxr\" (UniqueName: \"kubernetes.io/projected/2d4543d7-fc65-4c3c-8326-56fecd858313-kube-api-access-qmxxr\") pod \"payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr\" (UID: \"2d4543d7-fc65-4c3c-8326-56fecd858313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:05.544595 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.544554 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxxr\" (UniqueName: \"kubernetes.io/projected/2d4543d7-fc65-4c3c-8326-56fecd858313-kube-api-access-qmxxr\") pod \"payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr\" (UID: \"2d4543d7-fc65-4c3c-8326-56fecd858313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:05.544767 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.544699 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4543d7-fc65-4c3c-8326-56fecd858313-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr\" (UID: \"2d4543d7-fc65-4c3c-8326-56fecd858313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:05.545378 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.545359 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4543d7-fc65-4c3c-8326-56fecd858313-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr\" (UID: \"2d4543d7-fc65-4c3c-8326-56fecd858313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:05.553392 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.553370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxxr\" (UniqueName: \"kubernetes.io/projected/2d4543d7-fc65-4c3c-8326-56fecd858313-kube-api-access-qmxxr\") pod \"payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr\" (UID: \"2d4543d7-fc65-4c3c-8326-56fecd858313\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:05.737860 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.737822 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:05.879982 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:05.879952 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr"] Aug 01 01:24:05.882770 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:24:05.882739 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4543d7_fc65_4c3c_8326_56fecd858313.slice/crio-453e21a2586c932d94975134723bd7f3b6801c971beb69fa38618972fc1de0d6 WatchSource:0}: Error finding container 453e21a2586c932d94975134723bd7f3b6801c971beb69fa38618972fc1de0d6: Status 404 returned error can't find the container with id 453e21a2586c932d94975134723bd7f3b6801c971beb69fa38618972fc1de0d6 Aug 01 01:24:06.322685 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:06.322652 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" event={"ID":"2d4543d7-fc65-4c3c-8326-56fecd858313","Type":"ContainerStarted","Data":"e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa"} Aug 01 01:24:06.322867 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:06.322700 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" event={"ID":"2d4543d7-fc65-4c3c-8326-56fecd858313","Type":"ContainerStarted","Data":"453e21a2586c932d94975134723bd7f3b6801c971beb69fa38618972fc1de0d6"} Aug 01 01:24:06.322867 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:06.322730 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:06.336203 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:06.336143 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" podStartSLOduration=1.3361235329999999 podStartE2EDuration="1.336123533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:24:06.335099414 +0000 UTC m=+3024.544727869" watchObservedRunningTime="2026-08-01 01:24:06.336123533 +0000 UTC m=+3024.545751991" Aug 01 01:24:08.097842 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:08.097806 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:24:17.332048 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:17.332011 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:24:54.491630 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:24:54.491545 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:25:18.694390 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:25:18.694356 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:25:51.792756 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:25:51.792722 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:26:05.592656 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:26:05.592578 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:26:13.293684 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:26:13.293645 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:26:22.398901 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:26:22.398865 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:26:32.303706 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:26:32.303672 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:26:40.296838 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:26:40.296801 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:26:49.598719 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:26:49.598683 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:27:00.102088 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:27:00.102057 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:27:11.296320 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:27:11.296278 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:27:25.809168 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:27:25.809132 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:27:36.111999 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:27:36.111903 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:27:42.715280 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:27:42.715247 2568 scope.go:117] "RemoveContainer" containerID="f82816dba96ed3e901c3a2cb18c629f767854a8c15ca361747f88a1621d637a7" Aug 01 01:27:42.724939 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:27:42.724918 2568 scope.go:117] "RemoveContainer" containerID="389ecf6375576e817a78c36ab7d96edcc7f9b3dde05dfc75dcb7c0f78958da45" Aug 01 01:27:43.718156 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:27:43.718120 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:27:57.708827 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:27:57.708789 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:28:09.506417 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:09.506368 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:28:24.500837 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:24.500800 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:28:34.692676 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:34.692639 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:28:43.106017 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.105980 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78"] Aug 01 01:28:43.109848 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.109825 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.113445 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.113402 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"openshift-service-ca.crt\"" Aug 01 01:28:43.113569 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.113459 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"kube-root-ca.crt\"" Aug 01 01:28:43.113569 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.113465 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"sub-test-model-423d15-kserve-self-signed-certs\"" Aug 01 01:28:43.113569 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.113462 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-423d15\"/\"default-dockercfg-pslct\"" Aug 01 01:28:43.119926 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.119906 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78"] Aug 01 01:28:43.175543 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.175507 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cljj\" (UniqueName: \"kubernetes.io/projected/f2b16c5d-6f38-4862-97f5-a39ce5f82327-kube-api-access-4cljj\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.175543 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.175546 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tmp-dir\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.175767 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.175561 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tls-certs\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.175767 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.175681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-model-cache\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.175767 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.175725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-dshm\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.175871 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.175809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-home\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277007 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.276967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-model-cache\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277007 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.277016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-dshm\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277329 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.277055 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-home\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277329 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.277106 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cljj\" (UniqueName: \"kubernetes.io/projected/f2b16c5d-6f38-4862-97f5-a39ce5f82327-kube-api-access-4cljj\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277329 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.277126 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tmp-dir\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277329 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.277145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tls-certs\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277570 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.277528 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-model-cache\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277614 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.277564 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-home\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.277614 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.277606 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tmp-dir\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.279586 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.279557 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-dshm\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.279825 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.279809 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tls-certs\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.299910 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.299881 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cljj\" (UniqueName: \"kubernetes.io/projected/f2b16c5d-6f38-4862-97f5-a39ce5f82327-kube-api-access-4cljj\") pod \"sub-test-model-423d15-kserve-7f49f6bb89-k2n78\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.420726 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.420629 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:43.569820 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:43.569793 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78"] Aug 01 01:28:43.571552 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:28:43.571521 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2b16c5d_6f38_4862_97f5_a39ce5f82327.slice/crio-589f344d2f563a7bae52f5ceaba44ca765c98138f53d1110c892212b542228e9 WatchSource:0}: Error finding container 589f344d2f563a7bae52f5ceaba44ca765c98138f53d1110c892212b542228e9: Status 404 returned error can't find the container with id 589f344d2f563a7bae52f5ceaba44ca765c98138f53d1110c892212b542228e9 Aug 01 01:28:44.032430 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:44.032383 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" event={"ID":"f2b16c5d-6f38-4862-97f5-a39ce5f82327","Type":"ContainerStarted","Data":"6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513"} Aug 01 01:28:44.032430 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:44.032435 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" event={"ID":"f2b16c5d-6f38-4862-97f5-a39ce5f82327","Type":"ContainerStarted","Data":"589f344d2f563a7bae52f5ceaba44ca765c98138f53d1110c892212b542228e9"} Aug 01 01:28:44.050424 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:44.050370 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" podStartSLOduration=1.050354403 podStartE2EDuration="1.050354403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:28:44.047844578 +0000 UTC m=+3302.257473038" watchObservedRunningTime="2026-08-01 01:28:44.050354403 +0000 UTC m=+3302.259982859" Aug 01 01:28:45.590054 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:45.590016 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:28:53.421046 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:53.421009 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:53.421046 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:53.421058 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:53.434166 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:53.434141 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:28:54.109101 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:28:54.109066 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:29:19.097828 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:29:19.097780 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:29:42.762167 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:29:42.762131 2568 scope.go:117] "RemoveContainer" containerID="7964938b386a6c17a8ee510fcfd12ce09bf2bd1aed515a596b99fc9e5dfcbbdd" Aug 01 01:29:42.774094 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:29:42.774066 2568 scope.go:117] "RemoveContainer" containerID="bacbc5619ab3c6ce97f4f9e32f5dca85268d299b657b343fbd61159dd2c3bc8b" Aug 01 01:29:57.799321 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:29:57.799291 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:30:00.139668 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.139617 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx"] Aug 01 01:30:00.143500 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.143478 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx" Aug 01 01:30:00.144270 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.144243 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh"] Aug 01 01:30:00.145603 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.145582 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jfq8f\"" Aug 01 01:30:00.148269 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.148251 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh" Aug 01 01:30:00.154367 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.154335 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx"] Aug 01 01:30:00.157353 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.157322 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx" Aug 01 01:30:00.158846 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.158822 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh"] Aug 01 01:30:00.168709 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.168687 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh" Aug 01 01:30:00.355667 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.355638 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh"] Aug 01 01:30:00.358993 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:30:00.358962 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fad38f_0063_464f_918b_dcd9518cfe63.slice/crio-88944cbca36a74760a98c1349a64400c10273840ac7cce7352c0603b1c58fbd6 WatchSource:0}: Error finding container 88944cbca36a74760a98c1349a64400c10273840ac7cce7352c0603b1c58fbd6: Status 404 returned error can't find the container with id 88944cbca36a74760a98c1349a64400c10273840ac7cce7352c0603b1c58fbd6 Aug 01 01:30:00.519922 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.519846 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh" event={"ID":"a7fad38f-0063-464f-918b-dcd9518cfe63","Type":"ContainerStarted","Data":"88944cbca36a74760a98c1349a64400c10273840ac7cce7352c0603b1c58fbd6"} Aug 01 01:30:00.542771 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:00.542745 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx"] Aug 01 01:30:00.559556 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:30:00.559530 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667cfabd_cfdd_451b_9535_c6b068b534ad.slice/crio-4a9a5f13d23d75bc54c61aa377f2a6dcad900e0b2405d750b3a4cafe9525bf31 WatchSource:0}: Error finding container 4a9a5f13d23d75bc54c61aa377f2a6dcad900e0b2405d750b3a4cafe9525bf31: Status 404 returned error can't find the container with id 4a9a5f13d23d75bc54c61aa377f2a6dcad900e0b2405d750b3a4cafe9525bf31 Aug 01 01:30:01.535883 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:01.535853 2568 generic.go:356] "Generic (PLEG): container finished" podID="667cfabd-cfdd-451b-9535-c6b068b534ad" containerID="64edb572be7a72096302a026a15825430e785efe8b7f4948a4f6f772ecb36e11" exitCode=0 Aug 01 01:30:01.536337 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:01.535951 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx" event={"ID":"667cfabd-cfdd-451b-9535-c6b068b534ad","Type":"ContainerDied","Data":"64edb572be7a72096302a026a15825430e785efe8b7f4948a4f6f772ecb36e11"} Aug 01 01:30:01.536337 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:01.536041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx" event={"ID":"667cfabd-cfdd-451b-9535-c6b068b534ad","Type":"ContainerStarted","Data":"4a9a5f13d23d75bc54c61aa377f2a6dcad900e0b2405d750b3a4cafe9525bf31"} Aug 01 01:30:01.537445 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:01.537420 2568 generic.go:356] "Generic (PLEG): container finished" podID="a7fad38f-0063-464f-918b-dcd9518cfe63" containerID="b98cf2f20e8453f5a901823668bedb0cade1ba35ba1ef0f6b06a100647ccca8f" exitCode=0 Aug 01 01:30:01.537560 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:01.537452 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh" event={"ID":"a7fad38f-0063-464f-918b-dcd9518cfe63","Type":"ContainerDied","Data":"b98cf2f20e8453f5a901823668bedb0cade1ba35ba1ef0f6b06a100647ccca8f"} Aug 01 01:30:02.685519 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:02.685492 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx" Aug 01 01:30:02.720836 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:02.720804 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh" Aug 01 01:30:03.553643 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:03.553609 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx" Aug 01 01:30:03.553643 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:03.553635 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29759130-8j6qx" event={"ID":"667cfabd-cfdd-451b-9535-c6b068b534ad","Type":"ContainerDied","Data":"4a9a5f13d23d75bc54c61aa377f2a6dcad900e0b2405d750b3a4cafe9525bf31"} Aug 01 01:30:03.553891 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:03.553684 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a9a5f13d23d75bc54c61aa377f2a6dcad900e0b2405d750b3a4cafe9525bf31" Aug 01 01:30:03.554954 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:03.554922 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh" Aug 01 01:30:03.555106 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:03.554964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh" event={"ID":"a7fad38f-0063-464f-918b-dcd9518cfe63","Type":"ContainerDied","Data":"88944cbca36a74760a98c1349a64400c10273840ac7cce7352c0603b1c58fbd6"} Aug 01 01:30:03.555106 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:03.555003 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88944cbca36a74760a98c1349a64400c10273840ac7cce7352c0603b1c58fbd6" Aug 01 01:30:10.803271 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:10.803234 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:30:22.591328 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:22.591287 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:30:26.738883 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:26.738848 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78"] Aug 01 01:30:32.794403 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:32.794324 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:30:41.760462 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:41.760392 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" podUID="f2b16c5d-6f38-4862-97f5-a39ce5f82327" containerName="main" containerID="cri-o://6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513" gracePeriod=45 Aug 01 01:30:42.020054 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.019991 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:30:42.053453 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.053419 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-dshm\") pod \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " Aug 01 01:30:42.053603 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.053470 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tls-certs\") pod \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " Aug 01 01:30:42.053603 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.053549 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cljj\" (UniqueName: \"kubernetes.io/projected/f2b16c5d-6f38-4862-97f5-a39ce5f82327-kube-api-access-4cljj\") pod \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " Aug 01 01:30:42.053603 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.053574 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-home\") pod \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " Aug 01 01:30:42.053737 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.053626 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-model-cache\") pod \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " Aug 01 01:30:42.053737 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.053691 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tmp-dir\") pod \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\" (UID: \"f2b16c5d-6f38-4862-97f5-a39ce5f82327\") " Aug 01 01:30:42.053902 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.053851 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-home" (OuterVolumeSpecName: "home") pod "f2b16c5d-6f38-4862-97f5-a39ce5f82327" (UID: "f2b16c5d-6f38-4862-97f5-a39ce5f82327"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:30:42.054077 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.053935 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2b16c5d-6f38-4862-97f5-a39ce5f82327" (UID: "f2b16c5d-6f38-4862-97f5-a39ce5f82327"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:30:42.054077 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.054063 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2b16c5d-6f38-4862-97f5-a39ce5f82327" (UID: "f2b16c5d-6f38-4862-97f5-a39ce5f82327"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:30:42.055842 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.055824 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-dshm" (OuterVolumeSpecName: "dshm") pod "f2b16c5d-6f38-4862-97f5-a39ce5f82327" (UID: "f2b16c5d-6f38-4862-97f5-a39ce5f82327"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:30:42.055939 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.055921 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2b16c5d-6f38-4862-97f5-a39ce5f82327" (UID: "f2b16c5d-6f38-4862-97f5-a39ce5f82327"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:30:42.055986 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.055948 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b16c5d-6f38-4862-97f5-a39ce5f82327-kube-api-access-4cljj" (OuterVolumeSpecName: "kube-api-access-4cljj") pod "f2b16c5d-6f38-4862-97f5-a39ce5f82327" (UID: "f2b16c5d-6f38-4862-97f5-a39ce5f82327"). InnerVolumeSpecName "kube-api-access-4cljj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:30:42.154546 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.154517 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cljj\" (UniqueName: \"kubernetes.io/projected/f2b16c5d-6f38-4862-97f5-a39ce5f82327-kube-api-access-4cljj\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:30:42.154546 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.154544 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-home\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:30:42.154746 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.154555 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-model-cache\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:30:42.154746 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.154590 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tmp-dir\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:30:42.154746 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.154600 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b16c5d-6f38-4862-97f5-a39ce5f82327-dshm\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:30:42.154746 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.154611 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b16c5d-6f38-4862-97f5-a39ce5f82327-tls-certs\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:30:42.794687 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.794650 2568 generic.go:356] "Generic (PLEG): container finished" podID="f2b16c5d-6f38-4862-97f5-a39ce5f82327" containerID="6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513" exitCode=0 Aug 01 01:30:42.795390 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.794701 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" Aug 01 01:30:42.795390 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.794726 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" event={"ID":"f2b16c5d-6f38-4862-97f5-a39ce5f82327","Type":"ContainerDied","Data":"6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513"} Aug 01 01:30:42.795390 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.794774 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78" event={"ID":"f2b16c5d-6f38-4862-97f5-a39ce5f82327","Type":"ContainerDied","Data":"589f344d2f563a7bae52f5ceaba44ca765c98138f53d1110c892212b542228e9"} Aug 01 01:30:42.795390 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.794797 2568 scope.go:117] "RemoveContainer" containerID="6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513" Aug 01 01:30:42.805208 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.805187 2568 scope.go:117] "RemoveContainer" containerID="6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513" Aug 01 01:30:42.805637 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:30:42.805611 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513\": container with ID starting with 6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513 not found: ID does not exist" containerID="6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513" Aug 01 01:30:42.805716 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.805646 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513"} err="failed to get container status \"6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513\": rpc error: code = NotFound desc = could not find container \"6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513\": container with ID starting with 6596a725db2dc0ee3b62cf95b4bfc03a80cbf37fd1275361caa1d23201c0b513 not found: ID does not exist" Aug 01 01:30:42.810422 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.810399 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78"] Aug 01 01:30:42.813678 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:42.813656 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-423d15/sub-test-model-423d15-kserve-7f49f6bb89-k2n78"] Aug 01 01:30:44.231331 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:44.231299 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b16c5d-6f38-4862-97f5-a39ce5f82327" path="/var/lib/kubelet/pods/f2b16c5d-6f38-4862-97f5-a39ce5f82327/volumes" Aug 01 01:30:50.890182 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:30:50.890148 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:31:45.004397 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:31:45.004358 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:32:21.791881 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:21.791840 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:32:34.398824 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:34.398782 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:32:44.994282 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:44.994245 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:32:53.291107 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:53.291065 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:32:56.181520 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.181482 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-140095088564912"] Aug 01 01:32:56.181929 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.181917 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fad38f-0063-464f-918b-dcd9518cfe63" containerName="cleanup" Aug 01 01:32:56.181973 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.181930 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fad38f-0063-464f-918b-dcd9518cfe63" containerName="cleanup" Aug 01 01:32:56.181973 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.181952 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667cfabd-cfdd-451b-9535-c6b068b534ad" containerName="cleanup" Aug 01 01:32:56.181973 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.181959 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="667cfabd-cfdd-451b-9535-c6b068b534ad" containerName="cleanup" Aug 01 01:32:56.181973 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.181967 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b16c5d-6f38-4862-97f5-a39ce5f82327" containerName="main" Aug 01 01:32:56.181973 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.181973 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b16c5d-6f38-4862-97f5-a39ce5f82327" containerName="main" Aug 01 01:32:56.182130 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.182050 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b16c5d-6f38-4862-97f5-a39ce5f82327" containerName="main" Aug 01 01:32:56.182130 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.182060 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fad38f-0063-464f-918b-dcd9518cfe63" containerName="cleanup" Aug 01 01:32:56.182130 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.182067 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="667cfabd-cfdd-451b-9535-c6b068b534ad" containerName="cleanup" Aug 01 01:32:56.185306 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.185285 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088564912" Aug 01 01:32:56.187706 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.187681 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mvc7p\"" Aug 01 01:32:56.190731 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.190707 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088564912"] Aug 01 01:32:56.263733 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.263693 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kv4w\" (UniqueName: \"kubernetes.io/projected/3cacb94b-6e29-4005-a950-1d0367386060-kube-api-access-5kv4w\") pod \"test-curl-7652-140095088564912\" (UID: \"3cacb94b-6e29-4005-a950-1d0367386060\") " pod="opendatahub/test-curl-7652-140095088564912" Aug 01 01:32:56.364879 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.364827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kv4w\" (UniqueName: \"kubernetes.io/projected/3cacb94b-6e29-4005-a950-1d0367386060-kube-api-access-5kv4w\") pod \"test-curl-7652-140095088564912\" (UID: \"3cacb94b-6e29-4005-a950-1d0367386060\") " pod="opendatahub/test-curl-7652-140095088564912" Aug 01 01:32:56.372551 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.372523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kv4w\" (UniqueName: \"kubernetes.io/projected/3cacb94b-6e29-4005-a950-1d0367386060-kube-api-access-5kv4w\") pod \"test-curl-7652-140095088564912\" (UID: \"3cacb94b-6e29-4005-a950-1d0367386060\") " pod="opendatahub/test-curl-7652-140095088564912" Aug 01 01:32:56.498203 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.498118 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088564912" Aug 01 01:32:56.646788 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.646757 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088564912"] Aug 01 01:32:56.648161 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:32:56.648134 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cacb94b_6e29_4005_a950_1d0367386060.slice/crio-8749e08a6f3691d0334250463c67616273245bb60e062f30a647f5e56d6f3ddd WatchSource:0}: Error finding container 8749e08a6f3691d0334250463c67616273245bb60e062f30a647f5e56d6f3ddd: Status 404 returned error can't find the container with id 8749e08a6f3691d0334250463c67616273245bb60e062f30a647f5e56d6f3ddd Aug 01 01:32:56.649579 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:56.649561 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 01:32:57.632056 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:57.631968 2568 generic.go:356] "Generic (PLEG): container finished" podID="3cacb94b-6e29-4005-a950-1d0367386060" containerID="9673def8953aa4498b265375579bbf32183ff0503cecb0276ad62d149d009f97" exitCode=0 Aug 01 01:32:57.632429 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:57.632058 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088564912" event={"ID":"3cacb94b-6e29-4005-a950-1d0367386060","Type":"ContainerDied","Data":"9673def8953aa4498b265375579bbf32183ff0503cecb0276ad62d149d009f97"} Aug 01 01:32:57.632429 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:57.632103 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088564912" event={"ID":"3cacb94b-6e29-4005-a950-1d0367386060","Type":"ContainerStarted","Data":"8749e08a6f3691d0334250463c67616273245bb60e062f30a647f5e56d6f3ddd"} Aug 01 01:32:58.785720 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.785698 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088564912" Aug 01 01:32:58.817394 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.817367 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7652-140095088564912_3cacb94b-6e29-4005-a950-1d0367386060/test-curl-7652-140095088564912/0.log" Aug 01 01:32:58.844324 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.844290 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-140095088564912"] Aug 01 01:32:58.847650 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.847616 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-140095088564912"] Aug 01 01:32:58.888759 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.888728 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kv4w\" (UniqueName: \"kubernetes.io/projected/3cacb94b-6e29-4005-a950-1d0367386060-kube-api-access-5kv4w\") pod \"3cacb94b-6e29-4005-a950-1d0367386060\" (UID: \"3cacb94b-6e29-4005-a950-1d0367386060\") " Aug 01 01:32:58.891024 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.890968 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cacb94b-6e29-4005-a950-1d0367386060-kube-api-access-5kv4w" (OuterVolumeSpecName: "kube-api-access-5kv4w") pod "3cacb94b-6e29-4005-a950-1d0367386060" (UID: "3cacb94b-6e29-4005-a950-1d0367386060"). InnerVolumeSpecName "kube-api-access-5kv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:32:58.979876 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.979837 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-140095088652464"] Aug 01 01:32:58.980306 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.980291 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cacb94b-6e29-4005-a950-1d0367386060" containerName="test-curl-7652-140095088564912" Aug 01 01:32:58.980306 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.980307 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cacb94b-6e29-4005-a950-1d0367386060" containerName="test-curl-7652-140095088564912" Aug 01 01:32:58.980423 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.980408 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cacb94b-6e29-4005-a950-1d0367386060" containerName="test-curl-7652-140095088564912" Aug 01 01:32:58.983692 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.983672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088652464" Aug 01 01:32:58.989819 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.989795 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088652464"] Aug 01 01:32:58.990464 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:58.990429 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kv4w\" (UniqueName: \"kubernetes.io/projected/3cacb94b-6e29-4005-a950-1d0367386060-kube-api-access-5kv4w\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:32:59.091145 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:59.091103 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29qmk\" (UniqueName: \"kubernetes.io/projected/931bf21e-313d-4993-ba1c-122f4cae4a80-kube-api-access-29qmk\") pod \"test-curl-7652-140095088652464\" (UID: \"931bf21e-313d-4993-ba1c-122f4cae4a80\") " pod="opendatahub/test-curl-7652-140095088652464" Aug 01 01:32:59.192003 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:59.191968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qmk\" (UniqueName: \"kubernetes.io/projected/931bf21e-313d-4993-ba1c-122f4cae4a80-kube-api-access-29qmk\") pod \"test-curl-7652-140095088652464\" (UID: \"931bf21e-313d-4993-ba1c-122f4cae4a80\") " pod="opendatahub/test-curl-7652-140095088652464" Aug 01 01:32:59.199392 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:59.199367 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qmk\" (UniqueName: \"kubernetes.io/projected/931bf21e-313d-4993-ba1c-122f4cae4a80-kube-api-access-29qmk\") pod \"test-curl-7652-140095088652464\" (UID: \"931bf21e-313d-4993-ba1c-122f4cae4a80\") " pod="opendatahub/test-curl-7652-140095088652464" Aug 01 01:32:59.294862 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:59.294820 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088652464" Aug 01 01:32:59.432176 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:59.432150 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088652464"] Aug 01 01:32:59.433481 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:32:59.433452 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod931bf21e_313d_4993_ba1c_122f4cae4a80.slice/crio-9a4cec578b389b288f981eeaa3af0e9e739e46fb64eedb676ff1f28b7ce32273 WatchSource:0}: Error finding container 9a4cec578b389b288f981eeaa3af0e9e739e46fb64eedb676ff1f28b7ce32273: Status 404 returned error can't find the container with id 9a4cec578b389b288f981eeaa3af0e9e739e46fb64eedb676ff1f28b7ce32273 Aug 01 01:32:59.647697 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:59.647669 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088564912" Aug 01 01:32:59.647881 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:59.647703 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8749e08a6f3691d0334250463c67616273245bb60e062f30a647f5e56d6f3ddd" Aug 01 01:32:59.649057 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:32:59.649028 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088652464" event={"ID":"931bf21e-313d-4993-ba1c-122f4cae4a80","Type":"ContainerStarted","Data":"9a4cec578b389b288f981eeaa3af0e9e739e46fb64eedb676ff1f28b7ce32273"} Aug 01 01:32:59.657802 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:32:59.657777 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7652-140095088564912\" is forbidden: User \"system:node:ip-10-0-137-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-42.ec2.internal' and this object" podUID="3cacb94b-6e29-4005-a950-1d0367386060" pod="opendatahub/test-curl-7652-140095088564912" Aug 01 01:33:00.232157 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:00.232119 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cacb94b-6e29-4005-a950-1d0367386060" path="/var/lib/kubelet/pods/3cacb94b-6e29-4005-a950-1d0367386060/volumes" Aug 01 01:33:00.656757 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:00.656665 2568 generic.go:356] "Generic (PLEG): container finished" podID="931bf21e-313d-4993-ba1c-122f4cae4a80" containerID="b35ed470c6df7a92bfd91c269f538bca5d23c86d51eb8c3a199df55b208de603" exitCode=0 Aug 01 01:33:00.656910 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:00.656759 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088652464" event={"ID":"931bf21e-313d-4993-ba1c-122f4cae4a80","Type":"ContainerDied","Data":"b35ed470c6df7a92bfd91c269f538bca5d23c86d51eb8c3a199df55b208de603"} Aug 01 01:33:01.804645 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:01.804623 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088652464" Aug 01 01:33:01.822317 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:01.822286 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7652-140095088652464_931bf21e-313d-4993-ba1c-122f4cae4a80/test-curl-7652-140095088652464/0.log" Aug 01 01:33:01.847810 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:01.847787 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-140095088652464"] Aug 01 01:33:01.851108 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:01.851085 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-140095088652464"] Aug 01 01:33:01.916293 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:01.916200 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29qmk\" (UniqueName: \"kubernetes.io/projected/931bf21e-313d-4993-ba1c-122f4cae4a80-kube-api-access-29qmk\") pod \"931bf21e-313d-4993-ba1c-122f4cae4a80\" (UID: \"931bf21e-313d-4993-ba1c-122f4cae4a80\") " Aug 01 01:33:01.918473 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:01.918439 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931bf21e-313d-4993-ba1c-122f4cae4a80-kube-api-access-29qmk" (OuterVolumeSpecName: "kube-api-access-29qmk") pod "931bf21e-313d-4993-ba1c-122f4cae4a80" (UID: "931bf21e-313d-4993-ba1c-122f4cae4a80"). InnerVolumeSpecName "kube-api-access-29qmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:33:02.017706 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.017678 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29qmk\" (UniqueName: \"kubernetes.io/projected/931bf21e-313d-4993-ba1c-122f4cae4a80-kube-api-access-29qmk\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:33:02.025599 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.025573 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:02.026024 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.026011 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931bf21e-313d-4993-ba1c-122f4cae4a80" containerName="test-curl-7652-140095088652464" Aug 01 01:33:02.026074 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.026026 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="931bf21e-313d-4993-ba1c-122f4cae4a80" containerName="test-curl-7652-140095088652464" Aug 01 01:33:02.026115 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.026106 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="931bf21e-313d-4993-ba1c-122f4cae4a80" containerName="test-curl-7652-140095088652464" Aug 01 01:33:02.033288 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.030489 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:02.036853 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.035506 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:02.118561 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.118530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98zn\" (UniqueName: \"kubernetes.io/projected/d54f89c7-9aab-4c06-84c0-8383eff9a517-kube-api-access-t98zn\") pod \"test-curl-7652-140095088545056\" (UID: \"d54f89c7-9aab-4c06-84c0-8383eff9a517\") " pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:02.219272 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.219242 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98zn\" (UniqueName: \"kubernetes.io/projected/d54f89c7-9aab-4c06-84c0-8383eff9a517-kube-api-access-t98zn\") pod \"test-curl-7652-140095088545056\" (UID: \"d54f89c7-9aab-4c06-84c0-8383eff9a517\") " pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:02.227187 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.227163 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98zn\" (UniqueName: \"kubernetes.io/projected/d54f89c7-9aab-4c06-84c0-8383eff9a517-kube-api-access-t98zn\") pod \"test-curl-7652-140095088545056\" (UID: \"d54f89c7-9aab-4c06-84c0-8383eff9a517\") " pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:02.232540 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.232514 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931bf21e-313d-4993-ba1c-122f4cae4a80" path="/var/lib/kubelet/pods/931bf21e-313d-4993-ba1c-122f4cae4a80/volumes" Aug 01 01:33:02.347686 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.347651 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:02.484153 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.484123 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:02.485464 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:33:02.485438 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd54f89c7_9aab_4c06_84c0_8383eff9a517.slice/crio-30361045265abee610e0348280d13b1cda97c1b74de089ee4299923c14b3e9bd WatchSource:0}: Error finding container 30361045265abee610e0348280d13b1cda97c1b74de089ee4299923c14b3e9bd: Status 404 returned error can't find the container with id 30361045265abee610e0348280d13b1cda97c1b74de089ee4299923c14b3e9bd Aug 01 01:33:02.671550 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.671516 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088652464" Aug 01 01:33:02.671735 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.671559 2568 scope.go:117] "RemoveContainer" containerID="b35ed470c6df7a92bfd91c269f538bca5d23c86d51eb8c3a199df55b208de603" Aug 01 01:33:02.672980 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.672948 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088545056" event={"ID":"d54f89c7-9aab-4c06-84c0-8383eff9a517","Type":"ContainerStarted","Data":"30361045265abee610e0348280d13b1cda97c1b74de089ee4299923c14b3e9bd"} Aug 01 01:33:02.794536 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:02.794510 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:33:03.681358 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:03.681322 2568 generic.go:356] "Generic (PLEG): container finished" podID="d54f89c7-9aab-4c06-84c0-8383eff9a517" containerID="dd221e4e0f8773c4a192a41bfd310103b04a4eb5843f9c0d6149a5d93d6655c7" exitCode=0 Aug 01 01:33:03.681764 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:03.681369 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088545056" event={"ID":"d54f89c7-9aab-4c06-84c0-8383eff9a517","Type":"ContainerDied","Data":"dd221e4e0f8773c4a192a41bfd310103b04a4eb5843f9c0d6149a5d93d6655c7"} Aug 01 01:33:04.835405 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:04.835374 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:04.853629 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:04.853601 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7652-140095088545056_d54f89c7-9aab-4c06-84c0-8383eff9a517/test-curl-7652-140095088545056/0.log" Aug 01 01:33:04.879874 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:04.879838 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:04.882537 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:04.882515 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:04.944956 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:04.944870 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t98zn\" (UniqueName: \"kubernetes.io/projected/d54f89c7-9aab-4c06-84c0-8383eff9a517-kube-api-access-t98zn\") pod \"d54f89c7-9aab-4c06-84c0-8383eff9a517\" (UID: \"d54f89c7-9aab-4c06-84c0-8383eff9a517\") " Aug 01 01:33:04.947096 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:04.947068 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54f89c7-9aab-4c06-84c0-8383eff9a517-kube-api-access-t98zn" (OuterVolumeSpecName: "kube-api-access-t98zn") pod "d54f89c7-9aab-4c06-84c0-8383eff9a517" (UID: "d54f89c7-9aab-4c06-84c0-8383eff9a517"). InnerVolumeSpecName "kube-api-access-t98zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:33:05.020618 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.020583 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:05.021107 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.021087 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54f89c7-9aab-4c06-84c0-8383eff9a517" containerName="test-curl-7652-140095088545056" Aug 01 01:33:05.021228 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.021109 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54f89c7-9aab-4c06-84c0-8383eff9a517" containerName="test-curl-7652-140095088545056" Aug 01 01:33:05.021284 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.021244 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54f89c7-9aab-4c06-84c0-8383eff9a517" containerName="test-curl-7652-140095088545056" Aug 01 01:33:05.024331 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.024311 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:05.029374 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.029353 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:05.046606 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.046576 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98zn\" (UniqueName: \"kubernetes.io/projected/d54f89c7-9aab-4c06-84c0-8383eff9a517-kube-api-access-t98zn\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:33:05.148122 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.148081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9hv\" (UniqueName: \"kubernetes.io/projected/9577f79e-4538-4f17-b064-d42d27d9c824-kube-api-access-fn9hv\") pod \"test-curl-7652-140095088545056\" (UID: \"9577f79e-4538-4f17-b064-d42d27d9c824\") " pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:05.249195 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.249097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9hv\" (UniqueName: \"kubernetes.io/projected/9577f79e-4538-4f17-b064-d42d27d9c824-kube-api-access-fn9hv\") pod \"test-curl-7652-140095088545056\" (UID: \"9577f79e-4538-4f17-b064-d42d27d9c824\") " pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:05.256628 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.256598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9hv\" (UniqueName: \"kubernetes.io/projected/9577f79e-4538-4f17-b064-d42d27d9c824-kube-api-access-fn9hv\") pod \"test-curl-7652-140095088545056\" (UID: \"9577f79e-4538-4f17-b064-d42d27d9c824\") " pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:05.335933 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.335893 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:05.476317 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.476288 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:05.477885 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:33:05.477863 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9577f79e_4538_4f17_b064_d42d27d9c824.slice/crio-a0affeae764d8bc39ba095b769c9cce0a536b9124bd033234ea13d2d77748c78 WatchSource:0}: Error finding container a0affeae764d8bc39ba095b769c9cce0a536b9124bd033234ea13d2d77748c78: Status 404 returned error can't find the container with id a0affeae764d8bc39ba095b769c9cce0a536b9124bd033234ea13d2d77748c78 Aug 01 01:33:05.695147 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.695115 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:05.695147 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.695150 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30361045265abee610e0348280d13b1cda97c1b74de089ee4299923c14b3e9bd" Aug 01 01:33:05.696532 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.696507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088545056" event={"ID":"9577f79e-4538-4f17-b064-d42d27d9c824","Type":"ContainerStarted","Data":"a0affeae764d8bc39ba095b769c9cce0a536b9124bd033234ea13d2d77748c78"} Aug 01 01:33:05.709390 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:05.708424 2568 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7652-140095088545056" oldPodUID="d54f89c7-9aab-4c06-84c0-8383eff9a517" podUID="9577f79e-4538-4f17-b064-d42d27d9c824" Aug 01 01:33:06.231831 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:06.231799 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54f89c7-9aab-4c06-84c0-8383eff9a517" path="/var/lib/kubelet/pods/d54f89c7-9aab-4c06-84c0-8383eff9a517/volumes" Aug 01 01:33:06.704230 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:06.704179 2568 generic.go:356] "Generic (PLEG): container finished" podID="9577f79e-4538-4f17-b064-d42d27d9c824" containerID="c104d2c9229d21b9ea2b7b3a6940fce40289fca93ee014a650da6a8da415717a" exitCode=0 Aug 01 01:33:06.704410 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:06.704255 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088545056" event={"ID":"9577f79e-4538-4f17-b064-d42d27d9c824","Type":"ContainerDied","Data":"c104d2c9229d21b9ea2b7b3a6940fce40289fca93ee014a650da6a8da415717a"} Aug 01 01:33:07.863024 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:07.862995 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:07.882385 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:07.882353 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7652-140095088545056_9577f79e-4538-4f17-b064-d42d27d9c824/test-curl-7652-140095088545056/0.log" Aug 01 01:33:07.909457 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:07.909427 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:07.911926 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:07.911903 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-140095088545056"] Aug 01 01:33:07.974935 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:07.974860 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn9hv\" (UniqueName: \"kubernetes.io/projected/9577f79e-4538-4f17-b064-d42d27d9c824-kube-api-access-fn9hv\") pod \"9577f79e-4538-4f17-b064-d42d27d9c824\" (UID: \"9577f79e-4538-4f17-b064-d42d27d9c824\") " Aug 01 01:33:07.977055 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:07.977034 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9577f79e-4538-4f17-b064-d42d27d9c824-kube-api-access-fn9hv" (OuterVolumeSpecName: "kube-api-access-fn9hv") pod "9577f79e-4538-4f17-b064-d42d27d9c824" (UID: "9577f79e-4538-4f17-b064-d42d27d9c824"). InnerVolumeSpecName "kube-api-access-fn9hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:33:08.039511 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.039482 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:08.039971 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.039945 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9577f79e-4538-4f17-b064-d42d27d9c824" containerName="test-curl-7652-140095088545056" Aug 01 01:33:08.039971 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.039962 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9577f79e-4538-4f17-b064-d42d27d9c824" containerName="test-curl-7652-140095088545056" Aug 01 01:33:08.040155 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.040052 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9577f79e-4538-4f17-b064-d42d27d9c824" containerName="test-curl-7652-140095088545056" Aug 01 01:33:08.043288 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.043270 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:08.048338 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.048312 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:08.076494 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.076468 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9hv\" (UniqueName: \"kubernetes.io/projected/9577f79e-4538-4f17-b064-d42d27d9c824-kube-api-access-fn9hv\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:33:08.177999 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.177957 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgkzd\" (UniqueName: \"kubernetes.io/projected/b34eb6ef-6d3e-4e50-86c6-dde859d40606-kube-api-access-dgkzd\") pod \"test-curl-7652-140095088521552\" (UID: \"b34eb6ef-6d3e-4e50-86c6-dde859d40606\") " pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:08.231860 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.231788 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9577f79e-4538-4f17-b064-d42d27d9c824" path="/var/lib/kubelet/pods/9577f79e-4538-4f17-b064-d42d27d9c824/volumes" Aug 01 01:33:08.278857 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.278822 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgkzd\" (UniqueName: \"kubernetes.io/projected/b34eb6ef-6d3e-4e50-86c6-dde859d40606-kube-api-access-dgkzd\") pod \"test-curl-7652-140095088521552\" (UID: \"b34eb6ef-6d3e-4e50-86c6-dde859d40606\") " pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:08.286498 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.286475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgkzd\" (UniqueName: \"kubernetes.io/projected/b34eb6ef-6d3e-4e50-86c6-dde859d40606-kube-api-access-dgkzd\") pod \"test-curl-7652-140095088521552\" (UID: \"b34eb6ef-6d3e-4e50-86c6-dde859d40606\") " pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:08.353597 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.353569 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:08.491498 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.491471 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:08.492537 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:33:08.492510 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb34eb6ef_6d3e_4e50_86c6_dde859d40606.slice/crio-7fb7ec6c278627c933936f62ff2442d4e3300d1d794f84a82e991ac7effd925d WatchSource:0}: Error finding container 7fb7ec6c278627c933936f62ff2442d4e3300d1d794f84a82e991ac7effd925d: Status 404 returned error can't find the container with id 7fb7ec6c278627c933936f62ff2442d4e3300d1d794f84a82e991ac7effd925d Aug 01 01:33:08.716928 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.716889 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088521552" event={"ID":"b34eb6ef-6d3e-4e50-86c6-dde859d40606","Type":"ContainerStarted","Data":"7fb7ec6c278627c933936f62ff2442d4e3300d1d794f84a82e991ac7effd925d"} Aug 01 01:33:08.718166 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.718133 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088545056" Aug 01 01:33:08.718166 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:08.718163 2568 scope.go:117] "RemoveContainer" containerID="c104d2c9229d21b9ea2b7b3a6940fce40289fca93ee014a650da6a8da415717a" Aug 01 01:33:09.726941 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:09.726908 2568 generic.go:356] "Generic (PLEG): container finished" podID="b34eb6ef-6d3e-4e50-86c6-dde859d40606" containerID="30e2d98198bf73b88b975baf7f96ba2bb29c2ef89beb6294f170fc87ff5086d8" exitCode=0 Aug 01 01:33:09.727362 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:09.726997 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088521552" event={"ID":"b34eb6ef-6d3e-4e50-86c6-dde859d40606","Type":"ContainerDied","Data":"30e2d98198bf73b88b975baf7f96ba2bb29c2ef89beb6294f170fc87ff5086d8"} Aug 01 01:33:10.885460 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:10.885436 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:10.903734 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:10.903707 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7652-140095088521552_b34eb6ef-6d3e-4e50-86c6-dde859d40606/test-curl-7652-140095088521552/0.log" Aug 01 01:33:10.946170 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:10.946142 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:10.949705 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:10.949686 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:11.003512 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.003457 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgkzd\" (UniqueName: \"kubernetes.io/projected/b34eb6ef-6d3e-4e50-86c6-dde859d40606-kube-api-access-dgkzd\") pod \"b34eb6ef-6d3e-4e50-86c6-dde859d40606\" (UID: \"b34eb6ef-6d3e-4e50-86c6-dde859d40606\") " Aug 01 01:33:11.005625 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.005601 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34eb6ef-6d3e-4e50-86c6-dde859d40606-kube-api-access-dgkzd" (OuterVolumeSpecName: "kube-api-access-dgkzd") pod "b34eb6ef-6d3e-4e50-86c6-dde859d40606" (UID: "b34eb6ef-6d3e-4e50-86c6-dde859d40606"). InnerVolumeSpecName "kube-api-access-dgkzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:33:11.080654 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.080630 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:11.081097 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.081084 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34eb6ef-6d3e-4e50-86c6-dde859d40606" containerName="test-curl-7652-140095088521552" Aug 01 01:33:11.081139 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.081099 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34eb6ef-6d3e-4e50-86c6-dde859d40606" containerName="test-curl-7652-140095088521552" Aug 01 01:33:11.081177 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.081163 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34eb6ef-6d3e-4e50-86c6-dde859d40606" containerName="test-curl-7652-140095088521552" Aug 01 01:33:11.084321 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.084305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:11.089818 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.089798 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:11.104616 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.104590 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgkzd\" (UniqueName: \"kubernetes.io/projected/b34eb6ef-6d3e-4e50-86c6-dde859d40606-kube-api-access-dgkzd\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:33:11.205351 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.205322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6zt\" (UniqueName: \"kubernetes.io/projected/2c0d6999-28fe-4899-8c98-dcfd8cd29e37-kube-api-access-rb6zt\") pod \"test-curl-7652-140095088521552\" (UID: \"2c0d6999-28fe-4899-8c98-dcfd8cd29e37\") " pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:11.306314 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.306230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6zt\" (UniqueName: \"kubernetes.io/projected/2c0d6999-28fe-4899-8c98-dcfd8cd29e37-kube-api-access-rb6zt\") pod \"test-curl-7652-140095088521552\" (UID: \"2c0d6999-28fe-4899-8c98-dcfd8cd29e37\") " pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:11.313381 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.313353 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6zt\" (UniqueName: \"kubernetes.io/projected/2c0d6999-28fe-4899-8c98-dcfd8cd29e37-kube-api-access-rb6zt\") pod \"test-curl-7652-140095088521552\" (UID: \"2c0d6999-28fe-4899-8c98-dcfd8cd29e37\") " pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:11.395357 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.395325 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:11.742930 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.742907 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:11.743084 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.742936 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb7ec6c278627c933936f62ff2442d4e3300d1d794f84a82e991ac7effd925d" Aug 01 01:33:11.746979 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.746957 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:11.748853 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:33:11.748834 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c0d6999_28fe_4899_8c98_dcfd8cd29e37.slice/crio-08c8bfae82034ac216f1e71a96c4f668a2d4acf18734fa8c34be23da320143aa WatchSource:0}: Error finding container 08c8bfae82034ac216f1e71a96c4f668a2d4acf18734fa8c34be23da320143aa: Status 404 returned error can't find the container with id 08c8bfae82034ac216f1e71a96c4f668a2d4acf18734fa8c34be23da320143aa Aug 01 01:33:11.755014 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:11.754991 2568 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7652-140095088521552" oldPodUID="b34eb6ef-6d3e-4e50-86c6-dde859d40606" podUID="2c0d6999-28fe-4899-8c98-dcfd8cd29e37" Aug 01 01:33:12.233585 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:12.233551 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34eb6ef-6d3e-4e50-86c6-dde859d40606" path="/var/lib/kubelet/pods/b34eb6ef-6d3e-4e50-86c6-dde859d40606/volumes" Aug 01 01:33:12.750183 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:12.750146 2568 generic.go:356] "Generic (PLEG): container finished" podID="2c0d6999-28fe-4899-8c98-dcfd8cd29e37" containerID="db725a8f9142a0c5cf0f680faceccdbec641900d870e0caa6e3c66489d2f32a7" exitCode=0 Aug 01 01:33:12.750369 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:12.750238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088521552" event={"ID":"2c0d6999-28fe-4899-8c98-dcfd8cd29e37","Type":"ContainerDied","Data":"db725a8f9142a0c5cf0f680faceccdbec641900d870e0caa6e3c66489d2f32a7"} Aug 01 01:33:12.750369 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:12.750274 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-140095088521552" event={"ID":"2c0d6999-28fe-4899-8c98-dcfd8cd29e37","Type":"ContainerStarted","Data":"08c8bfae82034ac216f1e71a96c4f668a2d4acf18734fa8c34be23da320143aa"} Aug 01 01:33:13.898440 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:13.898415 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:13.919385 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:13.919360 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7652-140095088521552_2c0d6999-28fe-4899-8c98-dcfd8cd29e37/test-curl-7652-140095088521552/0.log" Aug 01 01:33:13.944602 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:13.944573 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:13.947147 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:13.947125 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-140095088521552"] Aug 01 01:33:14.033391 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.033300 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb6zt\" (UniqueName: \"kubernetes.io/projected/2c0d6999-28fe-4899-8c98-dcfd8cd29e37-kube-api-access-rb6zt\") pod \"2c0d6999-28fe-4899-8c98-dcfd8cd29e37\" (UID: \"2c0d6999-28fe-4899-8c98-dcfd8cd29e37\") " Aug 01 01:33:14.035574 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.035545 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0d6999-28fe-4899-8c98-dcfd8cd29e37-kube-api-access-rb6zt" (OuterVolumeSpecName: "kube-api-access-rb6zt") pod "2c0d6999-28fe-4899-8c98-dcfd8cd29e37" (UID: "2c0d6999-28fe-4899-8c98-dcfd8cd29e37"). InnerVolumeSpecName "kube-api-access-rb6zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:33:14.134167 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.134132 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6zt\" (UniqueName: \"kubernetes.io/projected/2c0d6999-28fe-4899-8c98-dcfd8cd29e37-kube-api-access-rb6zt\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:33:14.231763 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.231732 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c0d6999-28fe-4899-8c98-dcfd8cd29e37" path="/var/lib/kubelet/pods/2c0d6999-28fe-4899-8c98-dcfd8cd29e37/volumes" Aug 01 01:33:14.767351 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.767323 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-140095088521552" Aug 01 01:33:14.767543 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.767322 2568 scope.go:117] "RemoveContainer" containerID="db725a8f9142a0c5cf0f680faceccdbec641900d870e0caa6e3c66489d2f32a7" Aug 01 01:33:14.779478 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.779453 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd"] Aug 01 01:33:14.779949 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.779933 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0d6999-28fe-4899-8c98-dcfd8cd29e37" containerName="test-curl-7652-140095088521552" Aug 01 01:33:14.780025 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.779954 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0d6999-28fe-4899-8c98-dcfd8cd29e37" containerName="test-curl-7652-140095088521552" Aug 01 01:33:14.780089 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.780076 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0d6999-28fe-4899-8c98-dcfd8cd29e37" containerName="test-curl-7652-140095088521552" Aug 01 01:33:14.784727 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.784708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.787603 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.787586 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-de9aaa-openshift-default-dockercfg-2mdq8\"" Aug 01 01:33:14.793892 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.793862 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd"] Aug 01 01:33:14.940035 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.939994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz47k\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-kube-api-access-gz47k\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.940035 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.940039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-envoy\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.940609 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.940114 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-data\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.940609 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.940179 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-token\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.940609 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.940224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-podinfo\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.940609 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.940308 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istiod-ca-cert\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.940609 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.940337 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-certs\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.940609 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.940360 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-credential-socket\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:14.940609 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:14.940390 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-socket\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041341 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-socket\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041341 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz47k\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-kube-api-access-gz47k\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041585 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-envoy\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041585 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-data\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041585 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-token\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041585 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-podinfo\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041585 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041542 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istiod-ca-cert\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041585 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-certs\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041897 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041615 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-credential-socket\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.041897 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041717 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-socket\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.042010 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.041969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-credential-socket\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.042066 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.042002 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-data\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.042268 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.042245 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-certs\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.042604 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.042584 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istiod-ca-cert\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.044653 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.044624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-envoy\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.044772 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.044733 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-podinfo\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.051843 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.051814 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-token\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.052085 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.052067 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz47k\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-kube-api-access-gz47k\") pod \"e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.108203 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.108169 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:15.254711 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.254687 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd"] Aug 01 01:33:15.257545 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:33:15.257518 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eeeefc_cd68_4b51_8ecb_79a4fe9ae9de.slice/crio-a1ef26548934f644af57ea1ce1e2318a1719307c834bfeda816f8a29e0a486ef WatchSource:0}: Error finding container a1ef26548934f644af57ea1ce1e2318a1719307c834bfeda816f8a29e0a486ef: Status 404 returned error can't find the container with id a1ef26548934f644af57ea1ce1e2318a1719307c834bfeda816f8a29e0a486ef Aug 01 01:33:15.259592 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.259558 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 01:33:15.259680 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.259636 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 01:33:15.259738 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.259682 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 01:33:15.775329 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.775285 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" event={"ID":"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de","Type":"ContainerStarted","Data":"9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2"} Aug 01 01:33:15.775515 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.775343 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" event={"ID":"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de","Type":"ContainerStarted","Data":"a1ef26548934f644af57ea1ce1e2318a1719307c834bfeda816f8a29e0a486ef"} Aug 01 01:33:15.793634 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:15.793592 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podStartSLOduration=1.793575878 podStartE2EDuration="1.793575878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:33:15.79173839 +0000 UTC m=+3574.001366847" watchObservedRunningTime="2026-08-01 01:33:15.793575878 +0000 UTC m=+3574.003204416" Aug 01 01:33:16.108352 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:16.108270 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:16.113544 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:16.113522 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:16.783999 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:16.783969 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:16.784972 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:16.784950 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:19.466665 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.466628 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8"] Aug 01 01:33:19.470698 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.470676 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:19.473031 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.473009 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-de9aaa\"" Aug 01 01:33:19.473162 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.473125 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-de9aaa-dockercfg-n67fr\"" Aug 01 01:33:19.486277 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.485012 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8"] Aug 01 01:33:19.499466 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.499432 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96"] Aug 01 01:33:19.503770 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.503749 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:19.510038 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.509995 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96"] Aug 01 01:33:19.588040 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.588006 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6lq\" (UniqueName: \"kubernetes.io/projected/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-kube-api-access-ql6lq\") pod \"payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8\" (UID: \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:19.588256 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.588134 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8\" (UID: \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:19.689405 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.689361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6lq\" (UniqueName: \"kubernetes.io/projected/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-kube-api-access-ql6lq\") pod \"payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8\" (UID: \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:19.689608 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.689424 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96\" (UID: \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:19.689608 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.689443 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2n8d\" (UniqueName: \"kubernetes.io/projected/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-kube-api-access-p2n8d\") pod \"payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96\" (UID: \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:19.689608 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.689480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8\" (UID: \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:19.690265 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.690203 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8\" (UID: \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:19.696997 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.696977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6lq\" (UniqueName: \"kubernetes.io/projected/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-kube-api-access-ql6lq\") pod \"payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8\" (UID: \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:19.783800 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.783720 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:19.790258 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.790201 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96\" (UID: \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:19.790362 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.790270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2n8d\" (UniqueName: \"kubernetes.io/projected/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-kube-api-access-p2n8d\") pod \"payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96\" (UID: \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:19.790955 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.790933 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96\" (UID: \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:19.798420 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.798394 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2n8d\" (UniqueName: \"kubernetes.io/projected/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-kube-api-access-p2n8d\") pod \"payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96\" (UID: \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:19.817720 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.817697 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:19.985085 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:19.985057 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96"] Aug 01 01:33:19.985675 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:33:19.985649 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05f8b49_5a8a_4266_8d9a_f0d060db71ca.slice/crio-a222ae4f023599fada946fea8641e8fd2dd1beec99f8d6b33cacb9c8e976665b WatchSource:0}: Error finding container a222ae4f023599fada946fea8641e8fd2dd1beec99f8d6b33cacb9c8e976665b: Status 404 returned error can't find the container with id a222ae4f023599fada946fea8641e8fd2dd1beec99f8d6b33cacb9c8e976665b Aug 01 01:33:20.154641 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:33:20.154594 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bc710f_cf85_43a4_ba19_7f0e3e3a45a5.slice/crio-a22286d77d044f43811ed83a5deb33f338c704ccbaaa417d91af1877f256415f WatchSource:0}: Error finding container a22286d77d044f43811ed83a5deb33f338c704ccbaaa417d91af1877f256415f: Status 404 returned error can't find the container with id a22286d77d044f43811ed83a5deb33f338c704ccbaaa417d91af1877f256415f Aug 01 01:33:20.154788 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.154729 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8"] Aug 01 01:33:20.811401 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.811360 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" event={"ID":"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5","Type":"ContainerStarted","Data":"7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9"} Aug 01 01:33:20.811843 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.811515 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:20.811843 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.811537 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" event={"ID":"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5","Type":"ContainerStarted","Data":"a22286d77d044f43811ed83a5deb33f338c704ccbaaa417d91af1877f256415f"} Aug 01 01:33:20.812953 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.812931 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" event={"ID":"c05f8b49-5a8a-4266-8d9a-f0d060db71ca","Type":"ContainerStarted","Data":"77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383"} Aug 01 01:33:20.813017 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.812973 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" event={"ID":"c05f8b49-5a8a-4266-8d9a-f0d060db71ca","Type":"ContainerStarted","Data":"a222ae4f023599fada946fea8641e8fd2dd1beec99f8d6b33cacb9c8e976665b"} Aug 01 01:33:20.813017 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.812994 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:20.825631 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.825594 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" podStartSLOduration=1.825579925 podStartE2EDuration="1.825579925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:33:20.824295835 +0000 UTC m=+3579.033924286" watchObservedRunningTime="2026-08-01 01:33:20.825579925 +0000 UTC m=+3579.035208418" Aug 01 01:33:20.840462 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:20.840414 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" podStartSLOduration=1.840400878 podStartE2EDuration="1.840400878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:33:20.837456534 +0000 UTC m=+3579.047084991" watchObservedRunningTime="2026-08-01 01:33:20.840400878 +0000 UTC m=+3579.050029335" Aug 01 01:33:22.398992 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:22.398957 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:33:31.821408 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:31.821379 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:33:31.821923 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:31.821767 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:33:32.044961 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:32.044373 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": read tcp 10.132.0.2:57534->10.132.0.87:15021: read: connection reset by peer" start-of-body= Aug 01 01:33:32.044961 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:32.044426 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": read tcp 10.132.0.2:57534->10.132.0.87:15021: read: connection reset by peer" Aug 01 01:33:32.914030 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:32.913952 2568 generic.go:356] "Generic (PLEG): container finished" podID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerID="9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2" exitCode=0 Aug 01 01:33:32.914421 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:32.914029 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" event={"ID":"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de","Type":"ContainerDied","Data":"9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2"} Aug 01 01:33:32.914590 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:32.914573 2568 scope.go:117] "RemoveContainer" containerID="9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2" Aug 01 01:33:32.915715 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:32.915689 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 01:33:32.915789 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:32.915773 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 01:33:32.915826 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:32.915812 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 01 01:33:33.922760 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:33.922731 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" event={"ID":"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de","Type":"ContainerStarted","Data":"b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6"} Aug 01 01:33:34.108688 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:34.108652 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:34.108880 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:34.108858 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Aug 01 01:33:34.108932 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:34.108912 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Aug 01 01:33:35.109317 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:35.109284 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Aug 01 01:33:35.109722 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:35.109333 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Aug 01 01:33:36.109059 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:36.109022 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Aug 01 01:33:36.109299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:36.109082 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Aug 01 01:33:37.108509 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:37.108479 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Aug 01 01:33:37.108894 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:37.108534 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Aug 01 01:33:38.109550 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:38.109514 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Aug 01 01:33:38.109943 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:38.109568 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Aug 01 01:33:39.112483 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:39.112408 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:39.112884 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:39.112838 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:39.113507 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:39.113490 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:33:48.563866 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.563831 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84"] Aug 01 01:33:48.568940 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.568921 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:48.572172 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.572153 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9ba870\"" Aug 01 01:33:48.572494 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.572479 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9ba870-dockercfg-q8klj\"" Aug 01 01:33:48.579533 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.579509 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84"] Aug 01 01:33:48.693874 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.693843 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84\" (UID: \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:48.694080 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.693906 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9kt\" (UniqueName: \"kubernetes.io/projected/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-kube-api-access-cq9kt\") pod \"payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84\" (UID: \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:48.794991 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.794954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84\" (UID: \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:48.795162 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.795034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9kt\" (UniqueName: \"kubernetes.io/projected/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-kube-api-access-cq9kt\") pod \"payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84\" (UID: \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:48.795737 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.795719 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84\" (UID: \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:48.802011 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.801986 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9kt\" (UniqueName: \"kubernetes.io/projected/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-kube-api-access-cq9kt\") pod \"payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84\" (UID: \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:48.880567 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:48.880478 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:49.021051 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:49.021023 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84"] Aug 01 01:33:49.022414 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:33:49.022385 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1dcede_3464_4e0d_99fc_73ea14db6e9e.slice/crio-f7c8a0bbd3c42dce32f1a7d05ba1284849a662777a031a8c18f45af803fda0fd WatchSource:0}: Error finding container f7c8a0bbd3c42dce32f1a7d05ba1284849a662777a031a8c18f45af803fda0fd: Status 404 returned error can't find the container with id f7c8a0bbd3c42dce32f1a7d05ba1284849a662777a031a8c18f45af803fda0fd Aug 01 01:33:50.033890 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:50.033857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" event={"ID":"7d1dcede-3464-4e0d-99fc-73ea14db6e9e","Type":"ContainerStarted","Data":"4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1"} Aug 01 01:33:50.034287 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:50.033922 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:33:50.034287 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:50.033943 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" event={"ID":"7d1dcede-3464-4e0d-99fc-73ea14db6e9e","Type":"ContainerStarted","Data":"f7c8a0bbd3c42dce32f1a7d05ba1284849a662777a031a8c18f45af803fda0fd"} Aug 01 01:33:50.049277 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:50.049200 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" podStartSLOduration=2.049178741 podStartE2EDuration="2.049178741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:33:50.047692007 +0000 UTC m=+3608.257320464" watchObservedRunningTime="2026-08-01 01:33:50.049178741 +0000 UTC m=+3608.258807197" Aug 01 01:33:51.393721 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:33:51.393685 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:34:01.041933 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:34:01.041888 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" podUID="7d1dcede-3464-4e0d-99fc-73ea14db6e9e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.90:9004: connect: connection refused" Aug 01 01:34:11.041261 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:34:11.041225 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:34:18.969917 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:34:18.969879 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9wqfm_21d26da9-9d38-4531-8ad6-cff1582092b8/payload-processing/0.log" Aug 01 01:34:23.995033 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:34:23.994996 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:34:47.896510 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:34:47.896465 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:34:58.919566 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:34:58.919525 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96_c05f8b49-5a8a-4266-8d9a-f0d060db71ca/payload-processing/0.log" Aug 01 01:34:59.052848 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:34:59.052815 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9wqfm_21d26da9-9d38-4531-8ad6-cff1582092b8/payload-processing/0.log" Aug 01 01:35:11.497463 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:35:11.497373 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:35:40.093010 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:35:40.092966 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:36:01.111745 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:01.111702 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:36:03.071416 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.071379 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96"] Aug 01 01:36:03.071897 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.071597 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" podUID="c05f8b49-5a8a-4266-8d9a-f0d060db71ca" containerName="payload-processing" containerID="cri-o://77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383" gracePeriod=30 Aug 01 01:36:03.078762 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.078694 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8"] Aug 01 01:36:03.079271 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.079233 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" podUID="b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5" containerName="payload-pre-processing" containerID="cri-o://7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9" gracePeriod=30 Aug 01 01:36:03.370856 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.370825 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:36:03.410656 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.410625 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:36:03.451368 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.451330 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql6lq\" (UniqueName: \"kubernetes.io/projected/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-kube-api-access-ql6lq\") pod \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\" (UID: \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\") " Aug 01 01:36:03.451589 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.451386 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-plugins-config-volume\") pod \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\" (UID: \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\") " Aug 01 01:36:03.451589 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.451438 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2n8d\" (UniqueName: \"kubernetes.io/projected/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-kube-api-access-p2n8d\") pod \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\" (UID: \"c05f8b49-5a8a-4266-8d9a-f0d060db71ca\") " Aug 01 01:36:03.451589 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.451538 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-plugins-config-volume\") pod \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\" (UID: \"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5\") " Aug 01 01:36:03.451980 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.451947 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c05f8b49-5a8a-4266-8d9a-f0d060db71ca" (UID: "c05f8b49-5a8a-4266-8d9a-f0d060db71ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:36:03.452301 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.452056 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5" (UID: "b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:36:03.453822 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.453787 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-kube-api-access-ql6lq" (OuterVolumeSpecName: "kube-api-access-ql6lq") pod "b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5" (UID: "b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5"). InnerVolumeSpecName "kube-api-access-ql6lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:36:03.453822 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.453807 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-kube-api-access-p2n8d" (OuterVolumeSpecName: "kube-api-access-p2n8d") pod "c05f8b49-5a8a-4266-8d9a-f0d060db71ca" (UID: "c05f8b49-5a8a-4266-8d9a-f0d060db71ca"). InnerVolumeSpecName "kube-api-access-p2n8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:36:03.552570 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.552516 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6lq\" (UniqueName: \"kubernetes.io/projected/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-kube-api-access-ql6lq\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:03.552570 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.552564 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:03.552570 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.552573 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2n8d\" (UniqueName: \"kubernetes.io/projected/c05f8b49-5a8a-4266-8d9a-f0d060db71ca-kube-api-access-p2n8d\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:03.552570 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.552582 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:03.917081 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.917044 2568 generic.go:356] "Generic (PLEG): container finished" podID="b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5" containerID="7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9" exitCode=0 Aug 01 01:36:03.917291 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.917094 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" Aug 01 01:36:03.917291 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.917132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" event={"ID":"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5","Type":"ContainerDied","Data":"7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9"} Aug 01 01:36:03.917291 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.917189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8" event={"ID":"b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5","Type":"ContainerDied","Data":"a22286d77d044f43811ed83a5deb33f338c704ccbaaa417d91af1877f256415f"} Aug 01 01:36:03.917291 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.917243 2568 scope.go:117] "RemoveContainer" containerID="7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9" Aug 01 01:36:03.918723 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.918697 2568 generic.go:356] "Generic (PLEG): container finished" podID="c05f8b49-5a8a-4266-8d9a-f0d060db71ca" containerID="77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383" exitCode=0 Aug 01 01:36:03.918826 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.918739 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" Aug 01 01:36:03.918872 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.918851 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" event={"ID":"c05f8b49-5a8a-4266-8d9a-f0d060db71ca","Type":"ContainerDied","Data":"77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383"} Aug 01 01:36:03.918938 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.918920 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96" event={"ID":"c05f8b49-5a8a-4266-8d9a-f0d060db71ca","Type":"ContainerDied","Data":"a222ae4f023599fada946fea8641e8fd2dd1beec99f8d6b33cacb9c8e976665b"} Aug 01 01:36:03.928906 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.928877 2568 scope.go:117] "RemoveContainer" containerID="7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9" Aug 01 01:36:03.929349 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:36:03.929317 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9\": container with ID starting with 7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9 not found: ID does not exist" containerID="7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9" Aug 01 01:36:03.929455 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.929363 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9"} err="failed to get container status \"7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9\": rpc error: code = NotFound desc = could not find container \"7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9\": container with ID starting with 7498da547ad47af6ae74906b6d8a212242a55327560d1a3e590f22154413d4b9 not found: ID does not exist" Aug 01 01:36:03.929455 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.929391 2568 scope.go:117] "RemoveContainer" containerID="77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383" Aug 01 01:36:03.939940 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.939909 2568 scope.go:117] "RemoveContainer" containerID="77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383" Aug 01 01:36:03.940718 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:36:03.940450 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383\": container with ID starting with 77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383 not found: ID does not exist" containerID="77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383" Aug 01 01:36:03.940851 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.940759 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383"} err="failed to get container status \"77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383\": rpc error: code = NotFound desc = could not find container \"77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383\": container with ID starting with 77db43a5cb2e579a883058d02489d1b91ea98470f8b06c4ca5bf187ecd991383 not found: ID does not exist" Aug 01 01:36:03.944439 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.944401 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96"] Aug 01 01:36:03.947511 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.947481 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de9aaa-69bcf56c9b-kcm96"] Aug 01 01:36:03.955804 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.955766 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8"] Aug 01 01:36:03.959432 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:03.959399 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de9aaa-956659f6d-cdsf8"] Aug 01 01:36:04.231947 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:04.231912 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5" path="/var/lib/kubelet/pods/b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5/volumes" Aug 01 01:36:04.232338 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:04.232329 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c05f8b49-5a8a-4266-8d9a-f0d060db71ca" path="/var/lib/kubelet/pods/c05f8b49-5a8a-4266-8d9a-f0d060db71ca/volumes" Aug 01 01:36:09.689628 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:09.689577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:36:51.447737 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:51.447693 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd"] Aug 01 01:36:51.448291 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:51.447952 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" containerID="cri-o://b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6" gracePeriod=30 Aug 01 01:36:54.116008 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:54.115965 2568 patch_prober.go:28] interesting pod/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 01 01:36:54.116421 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:54.116039 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 01 01:36:56.701519 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.701495 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:36:56.798653 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798560 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-envoy\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.798653 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798616 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-credential-socket\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.798653 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798648 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-podinfo\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.798948 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798679 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-token\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.798948 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798707 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istiod-ca-cert\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.798948 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798750 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-certs\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.798948 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798776 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-data\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.798948 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798806 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-socket\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.798948 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.798844 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz47k\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-kube-api-access-gz47k\") pod \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\" (UID: \"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de\") " Aug 01 01:36:56.799292 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799012 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:36:56.799292 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799083 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:36:56.799292 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799197 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:36:56.799292 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799109 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:36:56.799522 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799310 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:36:56.799522 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799335 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-certs\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:56.799522 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799357 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-data\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:56.799522 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799371 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-credential-socket\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:56.799522 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.799385 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istiod-ca-cert\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:56.801153 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.801125 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:36:56.801289 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.801159 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 01 01:36:56.801289 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.801187 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:36:56.801289 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.801239 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-kube-api-access-gz47k" (OuterVolumeSpecName: "kube-api-access-gz47k") pod "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" (UID: "b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de"). InnerVolumeSpecName "kube-api-access-gz47k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:36:56.900009 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.899972 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-token\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:56.900009 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.900000 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-workload-socket\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:56.900009 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.900010 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz47k\" (UniqueName: \"kubernetes.io/projected/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-kube-api-access-gz47k\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:56.900009 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.900019 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-envoy\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:56.900321 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:56.900027 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de-istio-podinfo\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:36:57.274503 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.274470 2568 generic.go:356] "Generic (PLEG): container finished" podID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerID="b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6" exitCode=0 Aug 01 01:36:57.274677 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.274517 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" Aug 01 01:36:57.274677 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.274568 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" event={"ID":"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de","Type":"ContainerDied","Data":"b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6"} Aug 01 01:36:57.274677 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.274610 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd" event={"ID":"b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de","Type":"ContainerDied","Data":"a1ef26548934f644af57ea1ce1e2318a1719307c834bfeda816f8a29e0a486ef"} Aug 01 01:36:57.274677 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.274631 2568 scope.go:117] "RemoveContainer" containerID="b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6" Aug 01 01:36:57.285003 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.284984 2568 scope.go:117] "RemoveContainer" containerID="9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2" Aug 01 01:36:57.293915 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.293897 2568 scope.go:117] "RemoveContainer" containerID="b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6" Aug 01 01:36:57.294171 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:36:57.294152 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6\": container with ID starting with b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6 not found: ID does not exist" containerID="b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6" Aug 01 01:36:57.294252 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.294180 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6"} err="failed to get container status \"b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6\": rpc error: code = NotFound desc = could not find container \"b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6\": container with ID starting with b1a63eacdf287bfdd90cc954cc679c56432c1eaa4eb06f1c7ee42a26288ae8f6 not found: ID does not exist" Aug 01 01:36:57.294252 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.294196 2568 scope.go:117] "RemoveContainer" containerID="9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2" Aug 01 01:36:57.294591 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:36:57.294568 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2\": container with ID starting with 9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2 not found: ID does not exist" containerID="9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2" Aug 01 01:36:57.294659 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.294602 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2"} err="failed to get container status \"9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2\": rpc error: code = NotFound desc = could not find container \"9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2\": container with ID starting with 9d3d1544a4c432bae2294ca675a96c113cf5e71c3c6d3826adf19d7fdee9abb2 not found: ID does not exist" Aug 01 01:36:57.296106 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.296087 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd"] Aug 01 01:36:57.300995 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:57.300974 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-de9aaa-openshift-default-5bc9c4d744-8flhd"] Aug 01 01:36:58.231797 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:36:58.231764 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" path="/var/lib/kubelet/pods/b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de/volumes" Aug 01 01:37:02.323933 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.323898 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84"] Aug 01 01:37:02.324696 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.324194 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" podUID="7d1dcede-3464-4e0d-99fc-73ea14db6e9e" containerName="payload-pre-processing" containerID="cri-o://4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1" gracePeriod=30 Aug 01 01:37:02.614588 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.614558 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:37:02.758882 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.758844 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-plugins-config-volume\") pod \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\" (UID: \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\") " Aug 01 01:37:02.759049 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.758978 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq9kt\" (UniqueName: \"kubernetes.io/projected/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-kube-api-access-cq9kt\") pod \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\" (UID: \"7d1dcede-3464-4e0d-99fc-73ea14db6e9e\") " Aug 01 01:37:02.759261 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.759207 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d1dcede-3464-4e0d-99fc-73ea14db6e9e" (UID: "7d1dcede-3464-4e0d-99fc-73ea14db6e9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:37:02.761229 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.761197 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-kube-api-access-cq9kt" (OuterVolumeSpecName: "kube-api-access-cq9kt") pod "7d1dcede-3464-4e0d-99fc-73ea14db6e9e" (UID: "7d1dcede-3464-4e0d-99fc-73ea14db6e9e"). InnerVolumeSpecName "kube-api-access-cq9kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:37:02.860068 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.859982 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:37:02.860068 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:02.860011 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9kt\" (UniqueName: \"kubernetes.io/projected/7d1dcede-3464-4e0d-99fc-73ea14db6e9e-kube-api-access-cq9kt\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:37:03.317570 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.317537 2568 generic.go:356] "Generic (PLEG): container finished" podID="7d1dcede-3464-4e0d-99fc-73ea14db6e9e" containerID="4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1" exitCode=0 Aug 01 01:37:03.317748 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.317583 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" Aug 01 01:37:03.317748 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.317624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" event={"ID":"7d1dcede-3464-4e0d-99fc-73ea14db6e9e","Type":"ContainerDied","Data":"4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1"} Aug 01 01:37:03.317748 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.317676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84" event={"ID":"7d1dcede-3464-4e0d-99fc-73ea14db6e9e","Type":"ContainerDied","Data":"f7c8a0bbd3c42dce32f1a7d05ba1284849a662777a031a8c18f45af803fda0fd"} Aug 01 01:37:03.317748 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.317698 2568 scope.go:117] "RemoveContainer" containerID="4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1" Aug 01 01:37:03.327808 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.327629 2568 scope.go:117] "RemoveContainer" containerID="4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1" Aug 01 01:37:03.328044 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:37:03.327911 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1\": container with ID starting with 4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1 not found: ID does not exist" containerID="4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1" Aug 01 01:37:03.328044 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.327938 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1"} err="failed to get container status \"4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1\": rpc error: code = NotFound desc = could not find container \"4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1\": container with ID starting with 4ac932b157a2324296fece69fac5a5593da8e9032a791c6a39ee9e141aeda5b1 not found: ID does not exist" Aug 01 01:37:03.340199 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.340144 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84"] Aug 01 01:37:03.344385 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:03.344361 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ba870-779854c5bd-6sh84"] Aug 01 01:37:04.232056 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:04.232020 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1dcede-3464-4e0d-99fc-73ea14db6e9e" path="/var/lib/kubelet/pods/7d1dcede-3464-4e0d-99fc-73ea14db6e9e/volumes" Aug 01 01:37:04.699643 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:04.699603 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:37:19.829106 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:19.829068 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz"] Aug 01 01:37:19.829777 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:19.829340 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" podUID="c2506d8a-2de5-4b54-a1b1-cce4d01adb75" containerName="maas-api" containerID="cri-o://d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550" gracePeriod=30 Aug 01 01:37:20.097622 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.097592 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:37:20.119170 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.119135 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-openshift-service-ca\") pod \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " Aug 01 01:37:20.119356 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.119202 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-maas-api-tls\") pod \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " Aug 01 01:37:20.119356 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.119312 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrvj2\" (UniqueName: \"kubernetes.io/projected/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-kube-api-access-qrvj2\") pod \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\" (UID: \"c2506d8a-2de5-4b54-a1b1-cce4d01adb75\") " Aug 01 01:37:20.119541 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.119514 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2506d8a-2de5-4b54-a1b1-cce4d01adb75" (UID: "c2506d8a-2de5-4b54-a1b1-cce4d01adb75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:37:20.121666 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.121635 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2506d8a-2de5-4b54-a1b1-cce4d01adb75" (UID: "c2506d8a-2de5-4b54-a1b1-cce4d01adb75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:37:20.121666 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.121634 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-kube-api-access-qrvj2" (OuterVolumeSpecName: "kube-api-access-qrvj2") pod "c2506d8a-2de5-4b54-a1b1-cce4d01adb75" (UID: "c2506d8a-2de5-4b54-a1b1-cce4d01adb75"). InnerVolumeSpecName "kube-api-access-qrvj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:37:20.220167 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.220136 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-maas-api-tls\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:37:20.220167 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.220166 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrvj2\" (UniqueName: \"kubernetes.io/projected/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-kube-api-access-qrvj2\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:37:20.220401 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.220183 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2506d8a-2de5-4b54-a1b1-cce4d01adb75-openshift-service-ca\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:37:20.432046 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.432011 2568 generic.go:356] "Generic (PLEG): container finished" podID="c2506d8a-2de5-4b54-a1b1-cce4d01adb75" containerID="d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550" exitCode=0 Aug 01 01:37:20.432281 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.432074 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" Aug 01 01:37:20.432281 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.432098 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" event={"ID":"c2506d8a-2de5-4b54-a1b1-cce4d01adb75","Type":"ContainerDied","Data":"d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550"} Aug 01 01:37:20.432281 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.432185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz" event={"ID":"c2506d8a-2de5-4b54-a1b1-cce4d01adb75","Type":"ContainerDied","Data":"f7c0500f5858e6e28710508a1ed31a0847d2df0b9660ce06f211eb3f4c80d5ef"} Aug 01 01:37:20.432281 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.432248 2568 scope.go:117] "RemoveContainer" containerID="d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550" Aug 01 01:37:20.443440 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.443418 2568 scope.go:117] "RemoveContainer" containerID="d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550" Aug 01 01:37:20.443777 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:37:20.443754 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550\": container with ID starting with d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550 not found: ID does not exist" containerID="d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550" Aug 01 01:37:20.443853 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.443789 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550"} err="failed to get container status \"d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550\": rpc error: code = NotFound desc = could not find container \"d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550\": container with ID starting with d365421892731205f9960a7a4c1ad212bde94f13756f02ea880ccf078cdce550 not found: ID does not exist" Aug 01 01:37:20.456185 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.456154 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz"] Aug 01 01:37:20.460797 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:20.460747 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-423d15-56c75d7c8c-bg4gz"] Aug 01 01:37:22.201990 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:22.201955 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:37:22.232684 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:22.232656 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2506d8a-2de5-4b54-a1b1-cce4d01adb75" path="/var/lib/kubelet/pods/c2506d8a-2de5-4b54-a1b1-cce4d01adb75/volumes" Aug 01 01:37:37.439701 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.439663 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh"] Aug 01 01:37:37.442601 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.442569 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-65669d-29759130-ntrjh"] Aug 01 01:37:37.485502 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.485463 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr"] Aug 01 01:37:37.485775 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.485746 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" podUID="2d4543d7-fc65-4c3c-8326-56fecd858313" containerName="payload-pre-processing" containerID="cri-o://e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa" gracePeriod=30 Aug 01 01:37:37.756934 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.756905 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:37:37.894807 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.894770 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmxxr\" (UniqueName: \"kubernetes.io/projected/2d4543d7-fc65-4c3c-8326-56fecd858313-kube-api-access-qmxxr\") pod \"2d4543d7-fc65-4c3c-8326-56fecd858313\" (UID: \"2d4543d7-fc65-4c3c-8326-56fecd858313\") " Aug 01 01:37:37.895003 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.894865 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4543d7-fc65-4c3c-8326-56fecd858313-plugins-config-volume\") pod \"2d4543d7-fc65-4c3c-8326-56fecd858313\" (UID: \"2d4543d7-fc65-4c3c-8326-56fecd858313\") " Aug 01 01:37:37.895273 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.895208 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4543d7-fc65-4c3c-8326-56fecd858313-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d4543d7-fc65-4c3c-8326-56fecd858313" (UID: "2d4543d7-fc65-4c3c-8326-56fecd858313"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:37:37.897186 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.897154 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4543d7-fc65-4c3c-8326-56fecd858313-kube-api-access-qmxxr" (OuterVolumeSpecName: "kube-api-access-qmxxr") pod "2d4543d7-fc65-4c3c-8326-56fecd858313" (UID: "2d4543d7-fc65-4c3c-8326-56fecd858313"). InnerVolumeSpecName "kube-api-access-qmxxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:37:37.996522 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.996425 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4543d7-fc65-4c3c-8326-56fecd858313-plugins-config-volume\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:37:37.996522 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:37.996466 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxxr\" (UniqueName: \"kubernetes.io/projected/2d4543d7-fc65-4c3c-8326-56fecd858313-kube-api-access-qmxxr\") on node \"ip-10-0-137-42.ec2.internal\" DevicePath \"\"" Aug 01 01:37:38.231433 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.231401 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7fad38f-0063-464f-918b-dcd9518cfe63" path="/var/lib/kubelet/pods/a7fad38f-0063-464f-918b-dcd9518cfe63/volumes" Aug 01 01:37:38.550711 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.550675 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d4543d7-fc65-4c3c-8326-56fecd858313" containerID="e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa" exitCode=0 Aug 01 01:37:38.551208 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.550735 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" Aug 01 01:37:38.551208 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.550765 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" event={"ID":"2d4543d7-fc65-4c3c-8326-56fecd858313","Type":"ContainerDied","Data":"e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa"} Aug 01 01:37:38.551208 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.550862 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr" event={"ID":"2d4543d7-fc65-4c3c-8326-56fecd858313","Type":"ContainerDied","Data":"453e21a2586c932d94975134723bd7f3b6801c971beb69fa38618972fc1de0d6"} Aug 01 01:37:38.551208 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.550896 2568 scope.go:117] "RemoveContainer" containerID="e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa" Aug 01 01:37:38.560892 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.560873 2568 scope.go:117] "RemoveContainer" containerID="e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa" Aug 01 01:37:38.561187 ip-10-0-137-42 kubenswrapper[2568]: E0801 01:37:38.561166 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa\": container with ID starting with e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa not found: ID does not exist" containerID="e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa" Aug 01 01:37:38.561377 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.561196 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa"} err="failed to get container status \"e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa\": rpc error: code = NotFound desc = could not find container \"e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa\": container with ID starting with e95ae832b06f4a94cc4090c2b9030eb6d4c88819d9db152cbea6a00314ae12aa not found: ID does not exist" Aug 01 01:37:38.567033 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.567005 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr"] Aug 01 01:37:38.569836 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:38.569812 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-65669d-7669dfd748-mmzrr"] Aug 01 01:37:39.498230 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:39.498180 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nhj5q"] Aug 01 01:37:40.231686 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:40.231653 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4543d7-fc65-4c3c-8326-56fecd858313" path="/var/lib/kubelet/pods/2d4543d7-fc65-4c3c-8326-56fecd858313/volumes" Aug 01 01:37:42.900106 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:42.900072 2568 scope.go:117] "RemoveContainer" containerID="b98cf2f20e8453f5a901823668bedb0cade1ba35ba1ef0f6b06a100647ccca8f" Aug 01 01:37:50.739882 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:50.739852 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fd6cn_62a42240-2659-4035-aa15-78bc1d79c7b8/manager/0.log" Aug 01 01:37:50.865337 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:50.865309 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hpjdf_3e554008-4f64-43bb-9406-2677b287e547/manager/0.log" Aug 01 01:37:51.240543 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:51.240516 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6vffs_8a7adf51-7ecd-4cf0-b5b5-31fda87f0465/manager/0.log" Aug 01 01:37:51.637991 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:51.637917 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f88dcc48-sj8bg_241cfc9b-d570-474b-8f0e-1b0a38036189/copy-manifests/0.log" Aug 01 01:37:51.644638 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:51.644611 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f88dcc48-sj8bg_241cfc9b-d570-474b-8f0e-1b0a38036189/manager/0.log" Aug 01 01:37:52.420789 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:52.420761 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29759130-8j6qx_667cfabd-cfdd-451b-9535-c6b068b534ad/cleanup/0.log" Aug 01 01:37:53.532783 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:53.532748 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-9ztrl_5bb3d5c0-5d81-4284-b683-db5971436d70/manager/0.log" Aug 01 01:37:53.661956 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:53.661927 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-zqsrc_52712577-618b-4f22-ba81-b3b515bbd2a3/manager/0.log" Aug 01 01:37:53.800915 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:53.800823 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-cplxt_4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e/kuadrant-console-plugin/0.log" Aug 01 01:37:53.928719 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:53.928683 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-th6rv_a269c374-4cdd-445f-8459-2c71eb088c98/manager/0.log" Aug 01 01:37:54.060816 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:54.060744 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-nhj5q_181985b1-54d0-4e9f-9048-e8bd4c2c2e88/limitador/0.log" Aug 01 01:37:54.581248 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:54.581206 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv_2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f/istio-proxy/0.log" Aug 01 01:37:54.949009 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:54.948982 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7798dfdbd7-m5lph_b1c44826-c266-422d-94d6-658e0247af81/kube-auth-proxy/0.log" Aug 01 01:37:55.078646 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:55.078613 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-z99dn_f7871781-59fd-4cac-b923-1332ffb1785b/istio-proxy/1.log" Aug 01 01:37:55.346023 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:55.345925 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9wqfm_21d26da9-9d38-4531-8ad6-cff1582092b8/payload-processing/0.log" Aug 01 01:37:55.465542 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:55.465510 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64956b7558-rqmzr_5027f8b0-fdc9-48e1-bdd1-898dafd7fae2/router/0.log" Aug 01 01:37:55.879776 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:55.879748 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rwnrz_f8b977f4-b3e4-41ff-bf88-5a69fb1128f9/main/0.log" Aug 01 01:37:56.523168 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:37:56.523137 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-qj7gd_5d35dc51-5b10-48d7-98e0-f8f148122863/main/0.log" Aug 01 01:38:05.329922 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:05.329829 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d85hp_d3e36be9-8c7d-4886-b9d1-c3e2aaf5ed5f/global-pull-secret-syncer/0.log" Aug 01 01:38:05.479923 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:05.479891 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-75wgw_fe6a6f9e-b36a-45e7-9f1a-b8bf821dc03a/konnectivity-agent/0.log" Aug 01 01:38:05.539511 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:05.539471 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-42.ec2.internal_d78001921e659c8098fde0e8ca179c5e/haproxy/0.log" Aug 01 01:38:09.873819 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:09.873774 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-9ztrl_5bb3d5c0-5d81-4284-b683-db5971436d70/manager/0.log" Aug 01 01:38:09.897609 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:09.897579 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-zqsrc_52712577-618b-4f22-ba81-b3b515bbd2a3/manager/0.log" Aug 01 01:38:09.923602 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:09.923570 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-cplxt_4cb76ea7-400a-4ab1-9979-53b6b9fe0a0e/kuadrant-console-plugin/0.log" Aug 01 01:38:10.012335 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:10.012303 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-th6rv_a269c374-4cdd-445f-8459-2c71eb088c98/manager/0.log" Aug 01 01:38:10.045108 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:10.045070 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-nhj5q_181985b1-54d0-4e9f-9048-e8bd4c2c2e88/limitador/0.log" Aug 01 01:38:11.652492 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:11.652459 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cde15bfa-85a5-4c51-b671-a52e95534f8f/alertmanager/0.log" Aug 01 01:38:11.672572 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:11.672539 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cde15bfa-85a5-4c51-b671-a52e95534f8f/config-reloader/0.log" Aug 01 01:38:11.692694 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:11.692667 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cde15bfa-85a5-4c51-b671-a52e95534f8f/kube-rbac-proxy-web/0.log" Aug 01 01:38:11.714103 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:11.714041 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cde15bfa-85a5-4c51-b671-a52e95534f8f/kube-rbac-proxy/0.log" Aug 01 01:38:11.734363 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:11.734332 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cde15bfa-85a5-4c51-b671-a52e95534f8f/kube-rbac-proxy-metric/0.log" Aug 01 01:38:11.754196 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:11.754170 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cde15bfa-85a5-4c51-b671-a52e95534f8f/prom-label-proxy/0.log" Aug 01 01:38:11.773545 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:11.773513 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cde15bfa-85a5-4c51-b671-a52e95534f8f/init-config-reloader/0.log" Aug 01 01:38:11.814691 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:11.814652 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-f7lwh_b1fac243-00b1-4b7d-a1c9-5ba809a41789/cluster-monitoring-operator/0.log" Aug 01 01:38:12.138281 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.138250 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v745r_d25f9979-ce83-4d01-9b0b-aabb2ddc7c68/node-exporter/0.log" Aug 01 01:38:12.169332 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.169305 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v745r_d25f9979-ce83-4d01-9b0b-aabb2ddc7c68/kube-rbac-proxy/0.log" Aug 01 01:38:12.193554 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.193525 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v745r_d25f9979-ce83-4d01-9b0b-aabb2ddc7c68/init-textfile/0.log" Aug 01 01:38:12.564542 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.564512 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bdd84f7c5-2wn9d_26b55d53-5bdf-454e-939e-50521ca86047/telemeter-client/0.log" Aug 01 01:38:12.586754 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.586723 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bdd84f7c5-2wn9d_26b55d53-5bdf-454e-939e-50521ca86047/reload/0.log" Aug 01 01:38:12.610173 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.610146 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bdd84f7c5-2wn9d_26b55d53-5bdf-454e-939e-50521ca86047/kube-rbac-proxy/0.log" Aug 01 01:38:12.638270 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.638207 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84864f89c5-7zbl8_cda0c369-3d5b-47d2-a9e5-1962a873c20d/thanos-query/0.log" Aug 01 01:38:12.662490 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.662460 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84864f89c5-7zbl8_cda0c369-3d5b-47d2-a9e5-1962a873c20d/kube-rbac-proxy-web/0.log" Aug 01 01:38:12.685691 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.685659 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84864f89c5-7zbl8_cda0c369-3d5b-47d2-a9e5-1962a873c20d/kube-rbac-proxy/0.log" Aug 01 01:38:12.704773 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.704740 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84864f89c5-7zbl8_cda0c369-3d5b-47d2-a9e5-1962a873c20d/prom-label-proxy/0.log" Aug 01 01:38:12.725909 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.725882 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84864f89c5-7zbl8_cda0c369-3d5b-47d2-a9e5-1962a873c20d/kube-rbac-proxy-rules/0.log" Aug 01 01:38:12.744191 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:12.744165 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84864f89c5-7zbl8_cda0c369-3d5b-47d2-a9e5-1962a873c20d/kube-rbac-proxy-metrics/0.log" Aug 01 01:38:14.339204 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:14.339170 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-dx56d_ca0bbe23-8ace-42cd-8357-552e69aa9648/networking-console-plugin/0.log" Aug 01 01:38:15.478069 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.478037 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5db46f485b-kmh86_01116a7c-9330-4c8c-82d4-b4947ae0c42a/console/0.log" Aug 01 01:38:15.510010 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.509975 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-hzmfd_4960ec19-0b91-4513-9a88-a0d078a45ec8/download-server/0.log" Aug 01 01:38:15.811699 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.811597 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh"] Aug 01 01:38:15.812164 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812143 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5" containerName="payload-pre-processing" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812166 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5" containerName="payload-pre-processing" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812192 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4543d7-fc65-4c3c-8326-56fecd858313" containerName="payload-pre-processing" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812200 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4543d7-fc65-4c3c-8326-56fecd858313" containerName="payload-pre-processing" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812230 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c05f8b49-5a8a-4266-8d9a-f0d060db71ca" containerName="payload-processing" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812240 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c05f8b49-5a8a-4266-8d9a-f0d060db71ca" containerName="payload-processing" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812255 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812264 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812277 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2506d8a-2de5-4b54-a1b1-cce4d01adb75" containerName="maas-api" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812286 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2506d8a-2de5-4b54-a1b1-cce4d01adb75" containerName="maas-api" Aug 01 01:38:15.812299 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812301 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1dcede-3464-4e0d-99fc-73ea14db6e9e" containerName="payload-pre-processing" Aug 01 01:38:15.812784 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812310 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1dcede-3464-4e0d-99fc-73ea14db6e9e" containerName="payload-pre-processing" Aug 01 01:38:15.812784 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812423 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8bc710f-cf85-43a4-ba19-7f0e3e3a45a5" containerName="payload-pre-processing" Aug 01 01:38:15.812784 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812438 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4543d7-fc65-4c3c-8326-56fecd858313" containerName="payload-pre-processing" Aug 01 01:38:15.812784 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812455 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5eeeefc-cd68-4b51-8ecb-79a4fe9ae9de" containerName="istio-proxy" Aug 01 01:38:15.812784 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812464 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1dcede-3464-4e0d-99fc-73ea14db6e9e" containerName="payload-pre-processing" Aug 01 01:38:15.812784 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812474 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2506d8a-2de5-4b54-a1b1-cce4d01adb75" containerName="maas-api" Aug 01 01:38:15.812784 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.812484 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c05f8b49-5a8a-4266-8d9a-f0d060db71ca" containerName="payload-processing" Aug 01 01:38:15.815662 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.815636 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:15.818117 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.818090 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jdcjl\"/\"kube-root-ca.crt\"" Aug 01 01:38:15.818733 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.818716 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jdcjl\"/\"openshift-service-ca.crt\"" Aug 01 01:38:15.818733 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.818728 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jdcjl\"/\"default-dockercfg-dsx7t\"" Aug 01 01:38:15.824567 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.824538 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh"] Aug 01 01:38:15.967145 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.967101 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-lib-modules\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:15.967145 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.967145 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-podres\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:15.967444 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.967244 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-sys\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:15.967444 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.967302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-proc\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:15.967444 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:15.967362 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljff\" (UniqueName: \"kubernetes.io/projected/8606632b-19de-4b42-b116-ca23298cf15e-kube-api-access-fljff\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068277 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljff\" (UniqueName: \"kubernetes.io/projected/8606632b-19de-4b42-b116-ca23298cf15e-kube-api-access-fljff\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068459 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-lib-modules\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068459 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-podres\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068459 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068395 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-sys\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068459 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068430 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-proc\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068641 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068521 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-proc\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068641 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068531 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-podres\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068641 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068531 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-lib-modules\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.068641 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.068536 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8606632b-19de-4b42-b116-ca23298cf15e-sys\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.078060 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.078027 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljff\" (UniqueName: \"kubernetes.io/projected/8606632b-19de-4b42-b116-ca23298cf15e-kube-api-access-fljff\") pod \"perf-node-gather-daemonset-l24vh\" (UID: \"8606632b-19de-4b42-b116-ca23298cf15e\") " pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.127627 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.127591 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.231062 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.231034 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-jrrzg_976974b1-114c-4c8e-96ce-86a0a2fb7552/volume-data-source-validator/0.log" Aug 01 01:38:16.282694 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.282666 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh"] Aug 01 01:38:16.284827 ip-10-0-137-42 kubenswrapper[2568]: W0801 01:38:16.284799 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8606632b_19de_4b42_b116_ca23298cf15e.slice/crio-3d1e50357a69b9824a76394853b8482d2699330ec38f721fcca1132a02e5dcfb WatchSource:0}: Error finding container 3d1e50357a69b9824a76394853b8482d2699330ec38f721fcca1132a02e5dcfb: Status 404 returned error can't find the container with id 3d1e50357a69b9824a76394853b8482d2699330ec38f721fcca1132a02e5dcfb Aug 01 01:38:16.796718 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.796679 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" event={"ID":"8606632b-19de-4b42-b116-ca23298cf15e","Type":"ContainerStarted","Data":"0e8e8e41e0adb9cb605b5d2d48edba6a3afd28f14f7abbbd445bbb72614e97f6"} Aug 01 01:38:16.797139 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.796738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" event={"ID":"8606632b-19de-4b42-b116-ca23298cf15e","Type":"ContainerStarted","Data":"3d1e50357a69b9824a76394853b8482d2699330ec38f721fcca1132a02e5dcfb"} Aug 01 01:38:16.797139 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.796824 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:16.817494 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:16.817415 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" podStartSLOduration=1.817391737 podStartE2EDuration="1.817391737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 01:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:38:16.815809682 +0000 UTC m=+3875.025438138" watchObservedRunningTime="2026-08-01 01:38:16.817391737 +0000 UTC m=+3875.027020194" Aug 01 01:38:17.451852 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:17.451823 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpbm6_30615951-9c11-4b6b-9b0a-00813d2ec60e/dns/0.log" Aug 01 01:38:17.469949 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:17.469915 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpbm6_30615951-9c11-4b6b-9b0a-00813d2ec60e/kube-rbac-proxy/0.log" Aug 01 01:38:17.576764 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:17.576737 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4mkf_10cbd086-8ace-48e5-bda2-915b5ba02111/dns-node-resolver/0.log" Aug 01 01:38:18.142891 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:18.142854 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vm2k4_06e31806-756d-4782-9193-ac8ce656d17b/node-ca/0.log" Aug 01 01:38:19.050088 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:19.050054 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv_2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f/istio-proxy/0.log" Aug 01 01:38:19.260890 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:19.260860 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7798dfdbd7-m5lph_b1c44826-c266-422d-94d6-658e0247af81/kube-auth-proxy/0.log" Aug 01 01:38:19.287936 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:19.287910 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-z99dn_f7871781-59fd-4cac-b923-1332ffb1785b/istio-proxy/0.log" Aug 01 01:38:19.294517 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:19.294480 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-z99dn_f7871781-59fd-4cac-b923-1332ffb1785b/istio-proxy/1.log" Aug 01 01:38:19.360936 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:19.360847 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9wqfm_21d26da9-9d38-4531-8ad6-cff1582092b8/payload-processing/0.log" Aug 01 01:38:19.383439 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:19.383404 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64956b7558-rqmzr_5027f8b0-fdc9-48e1-bdd1-898dafd7fae2/router/0.log" Aug 01 01:38:20.210797 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:20.210760 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kvg6c_8cdf8609-6802-4dc9-b5b6-ca974534987b/serve-healthcheck-canary/0.log" Aug 01 01:38:20.673430 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:20.673341 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-zmdst_7e2afe34-c831-4376-a249-119e5eb68763/insights-operator/0.log" Aug 01 01:38:20.674326 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:20.674307 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-zmdst_7e2afe34-c831-4376-a249-119e5eb68763/insights-operator/1.log" Aug 01 01:38:20.693834 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:20.693804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cpp8_d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b/kube-rbac-proxy/0.log" Aug 01 01:38:20.714604 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:20.714573 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cpp8_d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b/exporter/0.log" Aug 01 01:38:20.737388 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:20.737356 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cpp8_d8ebc20c-a592-4fb3-b8fe-b4f2924d9d0b/extractor/0.log" Aug 01 01:38:22.822649 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:22.822620 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jdcjl/perf-node-gather-daemonset-l24vh" Aug 01 01:38:22.914876 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:22.914841 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fd6cn_62a42240-2659-4035-aa15-78bc1d79c7b8/manager/0.log" Aug 01 01:38:23.152665 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:23.152537 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hpjdf_3e554008-4f64-43bb-9406-2677b287e547/manager/0.log" Aug 01 01:38:23.457366 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:23.457330 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6vffs_8a7adf51-7ecd-4cf0-b5b5-31fda87f0465/manager/0.log" Aug 01 01:38:23.667669 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:23.667639 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f88dcc48-sj8bg_241cfc9b-d570-474b-8f0e-1b0a38036189/manager/0.log" Aug 01 01:38:23.686013 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:23.685984 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f88dcc48-sj8bg_241cfc9b-d570-474b-8f0e-1b0a38036189/copy-manifests/0.log" Aug 01 01:38:24.953703 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:24.953670 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jb79l_97c1001d-cf73-4cfd-84a0-e3159d18e674/openshift-lws-operator/0.log" Aug 01 01:38:26.118798 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:26.118764 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv_2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f/istio-proxy/0.log" Aug 01 01:38:26.309905 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:26.309879 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7798dfdbd7-m5lph_b1c44826-c266-422d-94d6-658e0247af81/kube-auth-proxy/0.log" Aug 01 01:38:26.325337 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:26.325309 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-z99dn_f7871781-59fd-4cac-b923-1332ffb1785b/istio-proxy/0.log" Aug 01 01:38:26.332248 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:26.332205 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-z99dn_f7871781-59fd-4cac-b923-1332ffb1785b/istio-proxy/1.log" Aug 01 01:38:26.372247 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:26.372147 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9wqfm_21d26da9-9d38-4531-8ad6-cff1582092b8/payload-processing/0.log" Aug 01 01:38:26.384310 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:26.384280 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64956b7558-rqmzr_5027f8b0-fdc9-48e1-bdd1-898dafd7fae2/router/0.log" Aug 01 01:38:27.773900 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:27.773811 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rj6qv_2e00ef10-1dd8-4ac3-a5ca-7aefbaeb190f/istio-proxy/0.log" Aug 01 01:38:27.965590 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:27.965563 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7798dfdbd7-m5lph_b1c44826-c266-422d-94d6-658e0247af81/kube-auth-proxy/0.log" Aug 01 01:38:27.983058 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:27.983022 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-z99dn_f7871781-59fd-4cac-b923-1332ffb1785b/istio-proxy/0.log" Aug 01 01:38:27.989792 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:27.989761 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-z99dn_f7871781-59fd-4cac-b923-1332ffb1785b/istio-proxy/1.log" Aug 01 01:38:28.035645 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:28.035560 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-9wqfm_21d26da9-9d38-4531-8ad6-cff1582092b8/payload-processing/0.log" Aug 01 01:38:28.053722 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:28.053696 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64956b7558-rqmzr_5027f8b0-fdc9-48e1-bdd1-898dafd7fae2/router/0.log" Aug 01 01:38:30.387102 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:30.387069 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-6gx8p_f7b5f441-b04a-4b1c-9b75-9bf091c54a21/kube-storage-version-migrator-operator/1.log" Aug 01 01:38:30.388291 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:30.388272 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-6gx8p_f7b5f441-b04a-4b1c-9b75-9bf091c54a21/kube-storage-version-migrator-operator/0.log" Aug 01 01:38:31.366930 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.366893 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6sqf4_8fb3d89c-6d48-445c-9726-4b33c252e881/kube-multus/0.log" Aug 01 01:38:31.390060 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.390028 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4khxh_78d38db1-2bcf-4778-8227-2956eec1f6f0/kube-multus-additional-cni-plugins/0.log" Aug 01 01:38:31.410244 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.410198 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4khxh_78d38db1-2bcf-4778-8227-2956eec1f6f0/egress-router-binary-copy/0.log" Aug 01 01:38:31.430919 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.430889 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4khxh_78d38db1-2bcf-4778-8227-2956eec1f6f0/cni-plugins/0.log" Aug 01 01:38:31.450471 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.450436 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4khxh_78d38db1-2bcf-4778-8227-2956eec1f6f0/bond-cni-plugin/0.log" Aug 01 01:38:31.470005 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.469979 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4khxh_78d38db1-2bcf-4778-8227-2956eec1f6f0/routeoverride-cni/0.log" Aug 01 01:38:31.492273 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.492245 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4khxh_78d38db1-2bcf-4778-8227-2956eec1f6f0/whereabouts-cni-bincopy/0.log" Aug 01 01:38:31.511974 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.511947 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4khxh_78d38db1-2bcf-4778-8227-2956eec1f6f0/whereabouts-cni/0.log" Aug 01 01:38:31.865744 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.865709 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-249m9_dcf38574-98e0-464c-8f21-bdb577e4b0ec/network-metrics-daemon/0.log" Aug 01 01:38:31.881820 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:31.881796 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-249m9_dcf38574-98e0-464c-8f21-bdb577e4b0ec/kube-rbac-proxy/0.log" Aug 01 01:38:32.952731 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:32.952694 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ptg24_0007ab3a-bc9a-4a12-9a80-baee7909713c/ovn-controller/0.log" Aug 01 01:38:32.972523 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:32.972485 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ptg24_0007ab3a-bc9a-4a12-9a80-baee7909713c/ovn-acl-logging/0.log" Aug 01 01:38:33.000990 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:33.000955 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ptg24_0007ab3a-bc9a-4a12-9a80-baee7909713c/kube-rbac-proxy-node/0.log" Aug 01 01:38:33.022445 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:33.022401 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ptg24_0007ab3a-bc9a-4a12-9a80-baee7909713c/kube-rbac-proxy-ovn-metrics/0.log" Aug 01 01:38:33.038897 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:33.038862 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ptg24_0007ab3a-bc9a-4a12-9a80-baee7909713c/northd/0.log" Aug 01 01:38:33.061647 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:33.061614 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ptg24_0007ab3a-bc9a-4a12-9a80-baee7909713c/nbdb/0.log" Aug 01 01:38:33.080493 ip-10-0-137-42 kubenswrapper[2568]: I0801 01:38:33.080463 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ptg24_0007ab3a-bc9a-4a12-9a80-baee7909713c/sbdb/0.log"